2024-03-29T11:06:54.978 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:06:54.978 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/orig.config.yaml 2024-03-29T11:06:55.013 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:06:55.093 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622695' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbbdhHQxBQkJRN769eMBmzh/0mCvr4Sar+m7eLmjdJTrB7PzcV4p69K2BYniUODFRvANxG2lols4JsHnC2ZZ3Q= smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNA5jl3+PWU8sr0KWl1Dye4j9g+m0x3sFQfImSX++M/iQdOsHG4erVwWf4OzwDEZKro2PkTqEYhDkrEndVGrY+Y= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:06:55.093 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:06:55.094 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:06:55.094 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:06:55.096 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:06:55.120 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:06:55.120 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:06:55.120 INFO:teuthology.packaging:ref: None 2024-03-29T11:06:55.121 INFO:teuthology.packaging:tag: None 2024-03-29T11:06:55.121 INFO:teuthology.packaging:branch: main 2024-03-29T11:06:55.121 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:55.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:06:55.369 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:06:55.371 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:06:55.377 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:06:55.377 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:06:55.383 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:06:55.397 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:06:55.403 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:06:55.429 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.598261', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbbdhHQxBQkJRN769eMBmzh/0mCvr4Sar+m7eLmjdJTrB7PzcV4p69K2BYniUODFRvANxG2lols4JsHnC2ZZ3Q='} 2024-03-29T11:06:55.446 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.596330', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNA5jl3+PWU8sr0KWl1Dye4j9g+m0x3sFQfImSX++M/iQdOsHG4erVwWf4OzwDEZKro2PkTqEYhDkrEndVGrY+Y='} 2024-03-29T11:06:55.446 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:06:55.451 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:06:55.451 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:06:55.451 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:06:55.523 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd5a18e50d0>, signals=[15]) 2024-03-29T11:06:55.523 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:06:55.529 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:06:55.530 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:06:55.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:55.637 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:06:55.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:55.724 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:06:55.730 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-03-29T11:06:55.748 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-03-29T11:06:55.748 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:55.806 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:55.807 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:55.807 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:55.807 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:55.807 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:55.807 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-03-29T11:06:55.828 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-03-29T11:06:55.844 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-03-29T11:06:55.844 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:55.901 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:55.902 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:55.902 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:55.902 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:55.902 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-03-29T11:06:55.919 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:06:55.930 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:06:55.937 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:06:55.938 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:55.940 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:55.956 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:06:55.963 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:06:55.963 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:55.997 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:56.116 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:06:56.174 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:06:56.191 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:06:56.191 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:06:56.191 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:56.192 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:56.208 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-03-29T11:06:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:56.209 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-29T11:06:56.265 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:56.265 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-361.el9.x86_64 2024-03-29T11:06:56.265 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-03-29T11:06:56.267 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:56.267 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-361.el9.x86_64 2024-03-29T11:06:56.267 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-03-29T11:06:58.395 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:06:58.502 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:06:59.968 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-29T11:07:00.203 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-29T11:07:02.067 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-29T11:07:03.543 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-29T11:07:03.835 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-29T11:07:07.285 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-03-29T11:07:08.011 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T11:07:09.553 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-03-29T11:07:11.090 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T11:07:11.588 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-29T11:07:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T11:07:14.240 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T11:07:16.246 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-29T11:07:18.578 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:07:18.579 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:07:18.680 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:07:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-03-29T11:07:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:07:18.690 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 76 M 2024-03-29T11:07:18.690 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 123 M 2024-03-29T11:07:18.690 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:07:18.848 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:07:19.262 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-29T11:07:19.348 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T11:07:19.973 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-03-29T11:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-29T11:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:07:21.527 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:07:21.527 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:07:21.619 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:07:21.628 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 76 M 2024-03-29T11:07:21.629 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 123 M 2024-03-29T11:07:21.630 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:07:21.805 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:07:22.255 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T11:07:23.022 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T11:07:23.305 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T11:07:23.309 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:23.309 INFO:teuthology.orchestra.run.smithi079.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-29T11:07:23.309 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:07:23.684 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:07:23.685 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:07:24.059 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:07:24.516 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:07:24.517 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:07:24.710 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:24.791 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:26.842 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:07:27.523 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:27.622 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:27.667 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:27.803 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:30.390 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:30.531 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:32.883 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:32.927 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:35.280 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:35.379 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:38.032 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:40.102 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:08.868 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:08.868 INFO:teuthology.orchestra.run.smithi052.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:08.868 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:08:08.914 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.145 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.145 INFO:teuthology.orchestra.run.smithi079.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:09.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:08:09.191 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.261 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.262 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:09.262 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:09.262 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:09.535 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:09.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:09.536 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:09.782 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:08:09.783 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:08:09.956 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:08:10.141 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:10.273 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:10.274 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.274 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.274 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:10.274 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:10.274 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:08:10.274 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:10.274 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:10.283 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:10.357 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:10.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.357 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-03-29T11:08:10.379 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:10.379 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.379 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.379 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:10.379 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:10.379 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:08:10.379 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:10.380 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:10.402 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-03-29T11:08:10.402 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-03-29T11:08:10.402 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:10.403 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:08:10.404 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:10.405 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:10.405 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:10.460 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:10.460 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.460 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-03-29T11:08:10.494 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 68.el9 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8264254 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:10.495 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:08:10.496 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:10.497 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:10.497 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:10.497 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:10.979 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-03-29T11:08:11.083 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-03-29T11:08:12.464 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:12.485 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-03-29T11:08:12.487 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-03-29T11:08:12.502 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.REkLAgbJnN 2024-03-29T11:08:12.502 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.REkLAgbJnN 2024-03-29T11:08:12.570 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.REkLAgbJnN 2024-03-29T11:08:12.579 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:12.611 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-03-29T11:08:12.613 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-29T11:08:12.629 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.XEe8Ut3X6b 2024-03-29T11:08:12.629 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XEe8Ut3X6b 2024-03-29T11:08:12.711 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.REkLAgbJnN is 6KB 2024-03-29T11:08:12.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.XEe8Ut3X6b 2024-03-29T11:08:12.760 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.REkLAgbJnN 2024-03-29T11:08:12.774 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:12.818 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.XEe8Ut3X6b is 6KB 2024-03-29T11:08:12.839 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.XEe8Ut3X6b 2024-03-29T11:08:12.845 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:12.845 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:12.845 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:12.846 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:12.847 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:12.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:12.930 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:12.930 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:12.930 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:12.930 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:12.931 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:12.992 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-03-29T11:08:13.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-03-29T11:08:43.004 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:08:43.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:08:43.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:43.079 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:08:43.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-29T11:08:43.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:01.343 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-29T11:09:01.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-03-29T11:09:10.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:09:10.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:10.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-29T11:09:10.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:13.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-29T11:09:25.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:09:25.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:43.839 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-29T11:09:58.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:09:58.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:59.261 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-29T11:09:59.728 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:09:59.728 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:09:59.729 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-03-29T11:09:59.742 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:09:59.743 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:09:59.743 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:09:59.743 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:00.744 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:10:00.744 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:00.847 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-03-29T11:10:00.873 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:10.601 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-29T11:10:22.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:22.979 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-29T11:10:23.327 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-29T11:10:23.328 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:23.328 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-29T11:10:23.342 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:23.343 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:23.343 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:23.343 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:24.343 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:10:24.344 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:24.423 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-03-29T11:10:24.459 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:24.460 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:10:24.467 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:10:24.467 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:24.470 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:24.488 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:10:24.507 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:10:24.514 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:10:24.514 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:24.527 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:24.596 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:10:24.605 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:10:24.606 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:24.608 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:24.639 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:24.648 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:24.653 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:24.662 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:24.664 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:10:24.671 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:10:24.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:24.697 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:24.734 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:10:24.743 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:10:24.743 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:24.767 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:24.790 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:24.864 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:24.916 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:24.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:25.042 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:10:25.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:25.111 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:25.152 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:25.224 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:25.267 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:25.313 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:10:25.313 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:25.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-03-29T11:10:25.386 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-03-29T11:10:25.431 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:25.966 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:10:25.973 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:10:25.974 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:10:25.998 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:10:26.006 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-03-29T11:10:26.058 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-03-29T11:10:26.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-03-29T11:10:26.122 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-03-29T11:10:26.123 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:10:26.124 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-03-29T11:10:26.149 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-03-29T11:10:26.149 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-03-29T11:10:26.174 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-03-29T11:10:26.174 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi079.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:10:26.175 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:26.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:26.247 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:10:26.247 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:10:26.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:10:26.342 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:10:26.351 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:10:26.367 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:10:26.368 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi079.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:14:53.304 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi079.front.sepia.ceph.com')] 2024-03-29T11:14:53.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-29T11:14:53.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:14:53.404 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-29T11:14:53.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-29T11:14:53.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:14:53.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:14:53.572 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-29T11:14:53.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T11:14:53.654 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:14:53.668 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:14:53.669 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:14:53.669 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:14:53.671 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:14:53.671 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:14:53.700 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:14:53.716 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:14:53.725 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:14:53.743 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:14:53.746 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-03-29T11:14:53.759 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-03-29T11:14:53.772 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-03-29T11:14:53.777 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:14:53.785 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-03-29T11:14:53.792 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:14:53.802 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:14:53.818 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:14:53.847 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:14:53.851 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:14:53.852 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-29T11:14:53.852 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.852 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.852 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.852 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.882 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:14:53.884 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:14:53.885 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-29T11:14:53.885 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.885 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.885 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.885 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:14:53.886 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:14:53.895 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:14:53.895 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:14:53.895 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:14:53.896 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:14:53.902 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:14:53.902 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:14:53.902 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:14:53.903 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:14:53.905 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:14:53.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:14:54.126 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:14:54.126 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:14:54.138 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:14:54.138 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:14:54.295 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:14:54.295 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:14:54.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:14:54.306 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:14:54.307 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:14:54.307 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:14:54.324 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:14:54.324 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:14:54.335 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:14:54.337 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:14:54.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:14:54.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:14:54.474 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:14:54.487 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:14:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-03-29T11:14:54.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-03-29T11:14:54.554 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-03-29T11:14:54.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-03-29T11:14:54.787 INFO:teuthology.orchestra.run.smithi052.stdout:74 files removed 2024-03-29T11:14:54.797 INFO:teuthology.orchestra.run.smithi079.stdout:74 files removed 2024-03-29T11:14:54.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-03-29T11:14:54.824 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-03-29T11:14:55.906 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 104 kB/s | 82 kB 00:00 2024-03-29T11:14:56.183 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 77 kB/s | 82 kB 00:01 2024-03-29T11:14:56.340 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T11:14:56.509 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 37 kB/s | 10 kB 00:00 2024-03-29T11:14:56.757 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:14:56.834 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-29T11:14:57.176 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:14:57.253 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:14:58.626 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T11:14:58.770 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-29T11:14:59.725 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T11:14:59.944 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-03-29T11:15:01.864 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T11:15:03.353 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-03-29T11:15:06.090 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T11:15:07.938 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:15:09.875 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:15:15.015 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-29T11:15:17.811 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:15:17.822 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:15:17.823 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:15:17.824 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:15:17.825 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:17.826 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:15:17.826 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:15:17.826 INFO:teuthology.orchestra.run.smithi079.stdout:Install 30 Packages 2024-03-29T11:15:17.826 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-03-29T11:15:17.826 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:17.826 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 58 M 2024-03-29T11:15:17.827 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:15:18.774 INFO:teuthology.orchestra.run.smithi079.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 6.7 MB/s | 5.3 MB 00:00 2024-03-29T11:15:18.832 INFO:teuthology.orchestra.run.smithi079.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T11:15:18.959 INFO:teuthology.orchestra.run.smithi079.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.7 MB/s | 742 kB 00:00 2024-03-29T11:15:19.066 INFO:teuthology.orchestra.run.smithi079.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.6 MB/s | 498 kB 00:00 2024-03-29T11:15:19.675 INFO:teuthology.orchestra.run.smithi079.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.4 MB/s | 5.1 MB 00:00 2024-03-29T11:15:19.775 INFO:teuthology.orchestra.run.smithi079.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T11:15:20.009 INFO:teuthology.orchestra.run.smithi079.stdout:(7/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 9.8 MB/s | 20 MB 00:02 2024-03-29T11:15:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 508 kB/s | 131 kB 00:00 2024-03-29T11:15:20.092 INFO:teuthology.orchestra.run.smithi079.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.9 MB/s | 162 kB 00:00 2024-03-29T11:15:20.126 INFO:teuthology.orchestra.run.smithi079.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 3.4 MB/s | 320 kB 00:00 2024-03-29T11:15:20.185 INFO:teuthology.orchestra.run.smithi079.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.2 MB/s | 298 kB 00:00 2024-03-29T11:15:20.210 INFO:teuthology.orchestra.run.smithi079.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:15:20.394 INFO:teuthology.orchestra.run.smithi079.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 507 kB/s | 106 kB 00:00 2024-03-29T11:15:20.419 INFO:teuthology.orchestra.run.smithi079.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 763 kB/s | 159 kB 00:00 2024-03-29T11:15:20.436 INFO:teuthology.orchestra.run.smithi079.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:15:20.570 INFO:teuthology.orchestra.run.smithi079.stdout:(16/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 9.7 MB 00:02 2024-03-29T11:15:20.603 INFO:teuthology.orchestra.run.smithi079.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-03-29T11:15:20.636 INFO:teuthology.orchestra.run.smithi079.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-03-29T11:15:20.662 INFO:teuthology.orchestra.run.smithi079.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 453 kB/s | 41 kB 00:00 2024-03-29T11:15:20.679 INFO:teuthology.orchestra.run.smithi079.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 834 kB/s | 35 kB 00:00 2024-03-29T11:15:20.704 INFO:teuthology.orchestra.run.smithi079.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:15:20.721 INFO:teuthology.orchestra.run.smithi079.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 788 kB/s | 46 kB 00:00 2024-03-29T11:15:21.021 INFO:teuthology.orchestra.run.smithi079.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 87 kB/s | 26 kB 00:00 2024-03-29T11:15:21.146 INFO:teuthology.orchestra.run.smithi079.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 661 kB/s | 309 kB 00:00 2024-03-29T11:15:21.163 INFO:teuthology.orchestra.run.smithi079.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-03-29T11:15:21.213 INFO:teuthology.orchestra.run.smithi079.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:15:21.331 INFO:teuthology.orchestra.run.smithi079.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-29T11:15:21.439 INFO:teuthology.orchestra.run.smithi079.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-03-29T11:15:21.481 INFO:teuthology.orchestra.run.smithi079.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-29T11:15:21.598 INFO:teuthology.orchestra.run.smithi079.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-29T11:15:21.848 INFO:teuthology.orchestra.run.smithi079.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-03-29T11:15:21.923 INFO:teuthology.orchestra.run.smithi079.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-03-29T11:15:21.930 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:21.930 INFO:teuthology.orchestra.run.smithi079.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T11:15:21.930 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:15:21.966 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:15:21.966 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:15:22.439 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:15:22.440 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:15:23.933 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:15:24.021 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:15:24.053 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:15:24.078 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:15:24.109 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:15:24.126 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:15:24.293 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:15:24.349 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:15:24.674 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:15:24.738 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:24.813 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:24.900 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:15:24.940 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:15:24.986 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:15:25.014 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:15:25.058 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:15:25.128 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:15:25.150 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:15:25.214 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:15:25.249 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:15:25.355 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:15:25.392 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:15:25.432 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:15:25.465 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:15:25.493 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:15:25.526 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:15:25.855 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:15:25.890 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:15:25.962 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:15:25.994 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:15:26.032 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:15:26.484 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:15:26.509 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:15:26.588 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:15:26.656 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:15:26.686 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:15:28.802 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:15:28.841 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:15:29.381 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:15:29.419 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:15:29.504 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:15:29.504 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:15:29.504 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:15:29.504 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:29.528 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:15:39.235 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:15:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-03-29T11:15:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-03-29T11:15:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-03-29T11:15:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-03-29T11:15:39.235 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-03-29T11:15:39.236 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-03-29T11:15:39.236 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-03-29T11:15:39.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:40.117 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:15:40.142 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:15:40.142 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:40.142 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:15:40.142 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:15:40.142 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:15:40.142 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:40.144 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:15:40.227 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:15:40.228 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:15:41.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:15:41.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:15:41.340 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:15:41.340 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:15:41.340 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:15:41.340 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:15:41.340 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:15:41.340 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:15:41.341 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:15:41.341 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:15:41.341 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:15:41.341 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.524 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.525 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:42.526 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:15:42.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-03-29T11:15:43.417 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:15:43.571 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:15:43.572 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 48 M 2024-03-29T11:15:43.573 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 204 M 2024-03-29T11:15:43.574 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:15:44.020 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:15:44.087 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-29T11:15:44.113 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 632 kB/s | 163 kB 00:00 2024-03-29T11:15:44.138 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:15:44.355 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 775 kB/s | 187 kB 00:00 2024-03-29T11:15:48.171 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 47 MB 00:04 2024-03-29T11:15:48.174 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:48.175 INFO:teuthology.orchestra.run.smithi079.stdout:Total 10 MB/s | 48 MB 00:04 2024-03-29T11:15:48.189 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:15:48.211 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:15:48.211 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:15:48.595 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:15:48.596 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:15:49.163 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:15:49.239 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:15:49.274 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:15:49.307 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:15:49.347 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:15:49.375 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:15:53.846 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:15:53.932 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:15:56.420 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:15:56.420 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:15:56.421 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:15:56.421 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:15:56.421 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:15:56.421 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:15:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:15:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:56.838 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:15:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:15:56.839 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:56.839 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:15:56.839 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:15:56.839 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:15:56.839 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:56.839 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:15:57.149 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-03-29T11:15:57.720 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:15:57.874 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:15:57.876 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-03-29T11:15:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:15:57.876 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-03-29T11:15:57.876 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:15:57.876 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:15:57.877 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:15:57.878 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout:Install 36 Packages 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:15:57.880 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 30 M 2024-03-29T11:15:57.880 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 107 M 2024-03-29T11:15:57.880 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:15:58.635 INFO:teuthology.orchestra.run.smithi079.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-29T11:15:58.802 INFO:teuthology.orchestra.run.smithi079.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-03-29T11:15:58.861 INFO:teuthology.orchestra.run.smithi079.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.5 MB/s | 2.1 MB 00:00 2024-03-29T11:15:58.920 INFO:teuthology.orchestra.run.smithi079.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.1 MB/s | 247 kB 00:00 2024-03-29T11:15:59.053 INFO:teuthology.orchestra.run.smithi079.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:15:59.095 INFO:teuthology.orchestra.run.smithi079.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:15:59.129 INFO:teuthology.orchestra.run.smithi079.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-29T11:15:59.162 INFO:teuthology.orchestra.run.smithi079.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:15:59.196 INFO:teuthology.orchestra.run.smithi079.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:15:59.238 INFO:teuthology.orchestra.run.smithi079.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:15:59.305 INFO:teuthology.orchestra.run.smithi079.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:15:59.339 INFO:teuthology.orchestra.run.smithi079.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:15:59.372 INFO:teuthology.orchestra.run.smithi079.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:15:59.406 INFO:teuthology.orchestra.run.smithi079.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:15:59.648 INFO:teuthology.orchestra.run.smithi079.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.6 MB/s | 4.7 MB 00:01 2024-03-29T11:15:59.749 INFO:teuthology.orchestra.run.smithi079.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-03-29T11:16:00.057 INFO:teuthology.orchestra.run.smithi079.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-29T11:16:00.082 INFO:teuthology.orchestra.run.smithi079.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 454 kB/s | 151 kB 00:00 2024-03-29T11:16:00.183 INFO:teuthology.orchestra.run.smithi079.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-29T11:16:00.199 INFO:teuthology.orchestra.run.smithi079.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 306 kB/s | 43 kB 00:00 2024-03-29T11:16:00.250 INFO:teuthology.orchestra.run.smithi079.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-29T11:16:00.275 INFO:teuthology.orchestra.run.smithi079.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:16:00.300 INFO:teuthology.orchestra.run.smithi079.stdout:(23/36): python3-requests-2.25.1-8.el9.noarch.r 140 kB/s | 125 kB 00:00 2024-03-29T11:16:00.318 INFO:teuthology.orchestra.run.smithi079.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2024-03-29T11:16:00.343 INFO:teuthology.orchestra.run.smithi079.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-29T11:16:00.368 INFO:teuthology.orchestra.run.smithi079.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-03-29T11:16:00.385 INFO:teuthology.orchestra.run.smithi079.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-29T11:16:00.419 INFO:teuthology.orchestra.run.smithi079.stdout:(28/36): python3-jaraco-functools-3.5.0-2.el9.n 164 kB/s | 19 kB 00:00 2024-03-29T11:16:00.453 INFO:teuthology.orchestra.run.smithi079.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:16:00.478 INFO:teuthology.orchestra.run.smithi079.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 721 kB/s | 79 kB 00:00 2024-03-29T11:16:00.503 INFO:teuthology.orchestra.run.smithi079.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 195 kB/s | 16 kB 00:00 2024-03-29T11:16:00.529 INFO:teuthology.orchestra.run.smithi079.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T11:16:00.554 INFO:teuthology.orchestra.run.smithi079.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-03-29T11:16:00.629 INFO:teuthology.orchestra.run.smithi079.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-29T11:16:00.654 INFO:teuthology.orchestra.run.smithi079.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-29T11:16:00.688 INFO:teuthology.orchestra.run.smithi079.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-29T11:16:00.692 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:00.692 INFO:teuthology.orchestra.run.smithi079.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T11:16:00.716 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:00.759 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:00.759 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:01.172 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:02.511 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:02.582 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:16:02.620 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:16:02.652 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:16:02.722 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:16:02.792 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:16:02.825 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:16:02.926 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:16:03.005 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:16:03.045 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:16:03.117 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:16:03.171 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:16:03.218 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:16:03.266 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:16:03.322 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:16:03.509 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:16:03.556 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:16:03.595 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:16:03.629 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:16:03.659 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:16:03.736 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:16:03.778 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:16:03.812 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:16:03.852 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:16:03.887 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:16:03.930 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:16:04.021 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:16:04.061 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:16:04.135 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:16:04.271 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:16:04.399 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:16:04.570 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:16:04.601 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:16:04.625 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:16:04.625 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:04.625 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:16:04.625 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:16:04.626 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:16:04.626 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:06.060 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:16:06.087 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:16:06.087 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:06.087 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:16:06.087 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:16:06.088 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:16:06.088 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:06.536 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:16:06.560 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:06.560 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:16:06.560 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:16:06.560 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:16:06.560 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:06.770 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:16:06.793 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:16:06.793 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:06.793 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:16:06.793 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:16:06.793 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:16:06.794 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:06.860 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:16:08.691 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:16:08.692 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:16:08.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:16:08.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:16:08.693 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:16:08.694 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:16:08.694 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:16:08.694 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:16:08.694 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:16:09.521 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:16:09.522 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:16:09.523 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:16:09.524 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:16:09.524 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:09.524 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:09.879 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-03-29T11:16:10.443 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:10.541 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:16:10.593 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:10.594 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:16:10.594 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:10.674 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-03-29T11:16:11.232 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:11.382 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:11.383 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:11.384 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:16:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:11.384 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 747 k 2024-03-29T11:16:11.384 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 754 k 2024-03-29T11:16:11.384 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:12.424 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 719 kB/s | 747 kB 00:01 2024-03-29T11:16:12.424 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:12.425 INFO:teuthology.orchestra.run.smithi079.stdout:Total 718 kB/s | 747 kB 00:01 2024-03-29T11:16:12.425 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:12.430 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:12.430 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:12.436 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:12.436 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:12.508 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:12.962 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:16:13.038 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:16:13.417 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:16:13.701 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:16:13.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:13.702 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:13.702 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:13.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:13.702 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:13.930 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:16:14.501 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:14.655 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:16:14.656 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:14.656 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 145 k 2024-03-29T11:16:14.656 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 436 k 2024-03-29T11:16:14.656 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:15.265 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 238 kB/s | 145 kB 00:00 2024-03-29T11:16:15.266 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:15.266 INFO:teuthology.orchestra.run.smithi079.stdout:Total 238 kB/s | 145 kB 00:00 2024-03-29T11:16:15.266 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:15.274 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:15.274 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:15.327 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:15.327 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:15.479 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:15.542 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:16:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:16:15.566 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:15.566 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:16:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:16.542 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:16:16.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:16.542 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:16.542 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:16.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:16.542 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:16.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-03-29T11:16:17.379 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:17.476 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:16:17.528 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:17.529 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:16:17.529 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:17.608 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:16:18.167 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:18.318 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:16:18.320 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-03-29T11:16:18.321 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:18.322 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.9 M 2024-03-29T11:16:18.322 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-03-29T11:16:18.322 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:18.863 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 68 kB/s | 15 kB 00:00 2024-03-29T11:16:18.905 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 117 kB/s | 31 kB 00:00 2024-03-29T11:16:19.072 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:16:19.114 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-29T11:16:19.330 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-03-29T11:16:19.397 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-29T11:16:19.639 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.4 MB/s | 2.0 MB 00:00 2024-03-29T11:16:19.712 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.0 MB/s | 144 kB 00:00 2024-03-29T11:16:19.756 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 693 kB/s | 31 kB 00:00 2024-03-29T11:16:19.815 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-29T11:16:20.074 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.2 MB/s | 1.8 MB 00:01 2024-03-29T11:16:20.799 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 500 kB/s | 863 kB 00:01 2024-03-29T11:16:20.805 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:20.805 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T11:16:20.825 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:20.843 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:20.843 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:21.023 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:21.023 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:21.528 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:21.607 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:16:21.643 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:16:21.689 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:16:21.721 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:16:21.825 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:16:21.946 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:16:22.040 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:16:22.084 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:16:22.113 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:16:22.145 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:16:22.379 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:16:22.411 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:16:23.173 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:16:23.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:16:23.549 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:23.549 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:23.549 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:16:23.550 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:23.551 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:23.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:16:24.376 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:24.528 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:24.529 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:16:24.530 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:24.531 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 37 M 2024-03-29T11:16:24.532 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-03-29T11:16:24.532 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:24.791 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 32 kB 00:00 2024-03-29T11:16:24.824 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 514 kB/s | 17 kB 00:00 2024-03-29T11:16:24.858 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:16:25.033 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-29T11:16:25.175 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-29T11:16:25.217 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-29T11:16:25.392 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-29T11:16:25.601 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-29T11:16:25.768 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:16:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:16:25.910 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:16:27.669 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.5 MB/s | 7.4 MB 00:03 2024-03-29T11:16:27.671 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:27.671 INFO:teuthology.orchestra.run.smithi079.stdout:Total 12 MB/s | 37 MB 00:03 2024-03-29T11:16:27.715 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:27.740 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:27.740 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:28.104 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:28.104 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:28.806 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:28.874 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:16:28.915 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:16:29.000 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:16:29.043 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:16:29.219 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:16:29.243 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 76 kB/s | 6.1 MB 01:22 2024-03-29T11:16:29.250 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:16:29.334 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:16:29.977 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:16:30.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:16:30.736 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-29T11:16:31.330 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:16:32.358 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:16:32.380 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:16:32.865 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-29T11:16:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:16:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:16:34.957 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:16:34.958 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:16:35.374 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:35.375 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:35.748 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:16:36.316 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:36.467 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:36.468 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T11:16:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:36.468 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T11:16:36.468 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:36.468 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:16:36.468 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:16:36.469 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:36.470 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:36.470 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T11:16:36.470 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-03-29T11:16:36.470 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:36.470 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.0 M 2024-03-29T11:16:36.470 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-03-29T11:16:36.471 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:36.657 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 244 kB/s | 24 kB 00:00 2024-03-29T11:16:36.716 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-29T11:16:36.741 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-03-29T11:16:36.767 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-29T11:16:36.792 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:16:36.809 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:16:37.109 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 95 kB/s | 52 kB 00:00 2024-03-29T11:16:37.210 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2024-03-29T11:16:37.252 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-03-29T11:16:37.452 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-03-29T11:16:37.619 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-03-29T11:16:37.744 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 319 kB/s | 203 kB 00:00 2024-03-29T11:16:37.803 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-29T11:16:37.807 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:37.808 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-29T11:16:37.808 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:37.821 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:37.821 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:37.914 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:37.914 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:37.971 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-03-29T11:16:38.421 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:38.574 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:16:38.629 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:16:38.694 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:16:38.739 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:16:38.780 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:16:38.883 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:16:39.000 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:16:39.049 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:16:39.767 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:16:39.822 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:16:39.865 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:16:39.912 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:16:39.934 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:16:40.658 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:40.673 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:40.674 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:40.675 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout:Install 30 Packages 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:16:40.676 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 58 M 2024-03-29T11:16:40.677 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:16:40.713 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:16:40.714 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:16:41.088 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:41.341 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:16:41.905 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:42.056 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:42.056 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T11:16:42.056 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:42.056 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-03-29T11:16:42.057 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:42.058 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.8 M 2024-03-29T11:16:42.058 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-03-29T11:16:42.058 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:42.457 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-03-29T11:16:42.533 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:16:42.649 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 495 kB/s | 58 kB 00:00 2024-03-29T11:16:42.691 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2024-03-29T11:16:42.708 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 878 kB/s | 51 kB 00:00 2024-03-29T11:16:42.975 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 188 kB/s | 138 kB 00:00 2024-03-29T11:16:42.978 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:42.978 INFO:teuthology.orchestra.run.smithi079.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2024-03-29T11:16:42.991 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:43.001 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:43.001 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:43.090 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:43.091 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:43.386 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:43.549 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:16:43.610 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:16:44.124 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:16:44.204 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:16:44.275 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:16:44.298 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:16:44.855 INFO:teuthology.orchestra.run.smithi052.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.4 MB/s | 9.7 MB 00:04 2024-03-29T11:16:44.955 INFO:teuthology.orchestra.run.smithi052.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-29T11:16:45.107 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:16:45.107 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:16:45.107 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:16:45.107 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:16:45.107 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:16:45.108 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:16:45.422 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:16:45.423 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:16:45.423 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:45.423 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:45.456 INFO:teuthology.orchestra.run.smithi052.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.4 MB/s | 742 kB 00:00 2024-03-29T11:16:45.679 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-03-29T11:16:45.848 INFO:teuthology.orchestra.run.smithi052.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.2 MB/s | 498 kB 00:00 2024-03-29T11:16:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:46.395 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:46.396 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:46.397 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-03-29T11:16:46.397 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:46.397 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 964 k 2024-03-29T11:16:46.397 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.8 M 2024-03-29T11:16:46.397 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:46.513 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 683 kB/s | 79 kB 00:00 2024-03-29T11:16:47.654 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 704 kB/s | 885 kB 00:01 2024-03-29T11:16:47.656 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:47.656 INFO:teuthology.orchestra.run.smithi079.stdout:Total 765 kB/s | 964 kB 00:01 2024-03-29T11:16:47.656 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:47.678 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:47.678 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:47.714 INFO:teuthology.orchestra.run.smithi052.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 786 kB/s | 5.3 MB 00:06 2024-03-29T11:16:47.753 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:47.753 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:47.806 INFO:teuthology.orchestra.run.smithi052.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-29T11:16:47.934 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:48.048 INFO:teuthology.orchestra.run.smithi052.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 543 kB/s | 131 kB 00:00 2024-03-29T11:16:48.081 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:16:48.155 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:16:48.373 INFO:teuthology.orchestra.run.smithi052.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 498 kB/s | 162 kB 00:00 2024-03-29T11:16:48.755 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:16:48.755 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:16:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 687 kB/s | 320 kB 00:00 2024-03-29T11:16:49.043 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:16:49.044 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:49.044 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:49.044 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:16:49.044 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:49.044 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:49.324 INFO:teuthology.orchestra.run.smithi052.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 616 kB/s | 298 kB 00:00 2024-03-29T11:16:49.327 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2024-03-29T11:16:49.499 INFO:teuthology.orchestra.run.smithi052.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 567 kB/s | 99 kB 00:00 2024-03-29T11:16:49.608 INFO:teuthology.orchestra.run.smithi052.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-29T11:16:49.683 INFO:teuthology.orchestra.run.smithi052.stdout:(13/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 5.1 MB 00:03 2024-03-29T11:16:49.708 INFO:teuthology.orchestra.run.smithi052.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-29T11:16:49.733 INFO:teuthology.orchestra.run.smithi052.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-29T11:16:49.775 INFO:teuthology.orchestra.run.smithi052.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:16:49.801 INFO:teuthology.orchestra.run.smithi052.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-29T11:16:49.846 INFO:teuthology.orchestra.run.smithi052.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 591 kB/s | 41 kB 00:00 2024-03-29T11:16:49.885 INFO:teuthology.orchestra.run.smithi052.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-03-29T11:16:49.900 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:49.910 INFO:teuthology.orchestra.run.smithi052.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 551 kB/s | 35 kB 00:00 2024-03-29T11:16:49.935 INFO:teuthology.orchestra.run.smithi052.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T11:16:50.057 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-03-29T11:16:50.059 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 332 k 2024-03-29T11:16:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.5 M 2024-03-29T11:16:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:50.178 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 580 kB/s | 69 kB 00:00 2024-03-29T11:16:50.277 INFO:teuthology.orchestra.run.smithi052.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 842 kB/s | 309 kB 00:00 2024-03-29T11:16:50.336 INFO:teuthology.orchestra.run.smithi052.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-29T11:16:50.403 INFO:teuthology.orchestra.run.smithi052.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 761 kB/s | 49 kB 00:00 2024-03-29T11:16:50.461 INFO:teuthology.orchestra.run.smithi052.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-03-29T11:16:50.487 INFO:teuthology.orchestra.run.smithi052.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-03-29T11:16:50.571 INFO:teuthology.orchestra.run.smithi052.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-03-29T11:16:50.596 INFO:teuthology.orchestra.run.smithi052.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-29T11:16:50.754 INFO:teuthology.orchestra.run.smithi052.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-29T11:16:51.661 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 165 kB/s | 264 kB 00:01 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi079.stdout:Total 207 kB/s | 332 kB 00:01 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:51.673 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:51.673 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:51.714 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:51.715 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:51.864 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:52.054 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:16:52.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:16:52.105 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:16:52.105 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:52.105 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:16:52.105 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:52.800 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:16:53.109 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:16:53.109 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:53.110 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:16:53.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:16:53.110 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:53.472 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-03-29T11:16:54.049 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:54.197 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:16:54.198 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:54.199 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 127 k 2024-03-29T11:16:54.199 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 456 k 2024-03-29T11:16:54.199 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:55.094 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 141 kB/s | 127 kB 00:00 2024-03-29T11:16:55.095 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:55.095 INFO:teuthology.orchestra.run.smithi079.stdout:Total 141 kB/s | 127 kB 00:00 2024-03-29T11:16:55.096 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:55.102 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:55.102 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:55.163 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:55.163 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:55.319 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:55.410 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:16:55.826 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:16:56.029 INFO:teuthology.orchestra.run.smithi052.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 636 kB/s | 3.4 MB 00:05 2024-03-29T11:16:56.109 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:16:56.109 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:56.109 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:56.109 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:56.109 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:56.109 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:56.392 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-03-29T11:16:56.966 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:57.065 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:16:57.121 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:57.122 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:16:57.123 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:57.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-03-29T11:16:57.783 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:16:57.934 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:16:57.935 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:16:57.937 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:16:57.938 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:57.938 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 31 k 2024-03-29T11:16:57.938 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 140 k 2024-03-29T11:16:57.938 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:16:58.416 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 65 kB/s | 31 kB 00:00 2024-03-29T11:16:58.416 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:58.417 INFO:teuthology.orchestra.run.smithi079.stdout:Total 65 kB/s | 31 kB 00:00 2024-03-29T11:16:58.417 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:16:58.420 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:16:58.420 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:16:58.454 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:16:58.454 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:16:58.588 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:16:58.681 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:16:59.050 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:16:59.323 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:16:59.323 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:59.323 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:16:59.323 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:59.323 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:16:59.323 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:16:59.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-03-29T11:16:59.679 INFO:teuthology.orchestra.run.smithi052.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 359 kB/s | 3.1 MB 00:08 2024-03-29T11:17:00.063 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:00.160 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:00.212 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:00.213 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:17:00.213 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:00.276 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-03-29T11:17:00.836 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:00.932 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:00.983 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:00.984 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:17:00.984 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:01.030 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-03-29T11:17:01.596 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:01.694 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:01.746 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:01.747 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:17:01.747 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:01.802 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-03-29T11:17:02.369 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:02.468 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:02.521 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:02.522 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:17:02.522 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:02.577 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-03-29T11:17:03.145 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:03.242 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:03.294 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:03.295 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:17:03.295 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:03.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-03-29T11:17:03.917 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:03.937 INFO:teuthology.orchestra.run.smithi052.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 878 kB/s | 20 MB 00:23 2024-03-29T11:17:03.943 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:03.943 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.5 MB/s | 58 MB 00:23 2024-03-29T11:17:03.943 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:17:03.985 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:17:03.985 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:17:04.015 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:04.067 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:04.068 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T11:17:04.068 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:04.111 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-03-29T11:17:04.472 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:17:04.472 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:17:04.671 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:04.820 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:04.821 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:04.821 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-03-29T11:17:04.821 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:04.821 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 85 k 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 231 k 2024-03-29T11:17:04.822 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:17:05.235 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 206 kB/s | 85 kB 00:00 2024-03-29T11:17:05.236 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:05.236 INFO:teuthology.orchestra.run.smithi079.stdout:Total 206 kB/s | 85 kB 00:00 2024-03-29T11:17:05.236 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:17:05.243 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:17:05.243 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:17:05.298 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:17:05.299 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:17:05.437 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:17:05.533 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:05.914 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:06.051 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:17:06.129 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:17:06.166 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:17:06.192 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:17:06.197 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:06.197 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:06.197 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:17:06.197 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:06.197 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:06.197 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:06.215 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:06.231 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:06.411 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-03-29T11:17:06.424 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:06.479 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:06.822 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:17:06.986 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:06.987 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:07.062 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:07.144 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:07.145 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:17:07.146 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:07.146 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.1 M 2024-03-29T11:17:07.146 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 13 M 2024-03-29T11:17:07.146 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:17:07.152 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:07.296 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:07.343 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:07.375 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:07.424 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:07.506 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:07.531 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:07.598 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:07.633 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:07.759 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:07.797 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:07.838 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:07.871 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:07.900 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:07.924 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:08.257 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:08.287 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:08.361 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:08.392 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:08.438 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:08.917 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:08.948 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:09.019 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:09.089 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:09.125 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:09.340 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-29T11:17:09.341 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:09.342 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-03-29T11:17:09.342 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:17:09.352 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:17:09.353 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:17:09.429 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:17:09.429 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:17:09.859 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:17:09.908 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:09.932 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:09.932 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:09.932 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:17:09.933 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:17:09.933 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:17:09.933 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:11.090 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:17:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:11.090 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:11.355 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-03-29T11:17:11.402 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:11.555 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:11.931 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:15:15 AM UTC. 2024-03-29T11:17:12.078 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T11:17:12.078 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 171 k 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 494 k 2024-03-29T11:17:12.079 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T11:17:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:12.184 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:12.266 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:12.266 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:12.267 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:12.267 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:12.310 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:12.545 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 368 kB/s | 171 kB 00:00 2024-03-29T11:17:12.545 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:12.545 INFO:teuthology.orchestra.run.smithi079.stdout:Total 367 kB/s | 171 kB 00:00 2024-03-29T11:17:12.546 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T11:17:12.553 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T11:17:12.554 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T11:17:12.609 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T11:17:12.609 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T11:17:12.756 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T11:17:12.876 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:13.270 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:13.544 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T11:17:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:13.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:17:13.544 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T11:17:13.799 DEBUG:teuthology.parallel:result is None 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:22.967 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:22.991 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:22.991 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:22.991 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:22.992 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:22.992 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:22.992 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:22.994 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:23.078 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:23.079 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:24.212 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:24.213 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:24.213 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:24.213 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:24.213 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:24.213 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:24.213 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:24.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:24.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:24.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:24.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:24.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:24.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:24.215 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:24.216 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:24.217 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:24.217 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:24.217 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:24.218 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:24.218 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:24.218 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:25.154 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:25.154 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:25.155 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.156 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:25.157 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:25.157 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:17:25.521 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-03-29T11:17:26.091 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:49 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:17:26.248 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:17:26.249 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-03-29T11:17:26.250 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:17:26.251 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 48 M 2024-03-29T11:17:26.251 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 204 M 2024-03-29T11:17:26.251 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:17:26.493 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2024-03-29T11:17:26.560 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:17:26.593 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:17:26.793 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 936 kB/s | 187 kB 00:00 2024-03-29T11:17:26.885 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 316 kB/s | 163 kB 00:00 2024-03-29T11:17:57.224 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 47 MB 00:30 2024-03-29T11:17:57.227 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:57.227 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.6 MB/s | 48 MB 00:30 2024-03-29T11:17:57.240 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:17:57.262 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:17:57.262 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:17:57.664 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:17:57.664 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:17:58.253 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:17:58.318 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:58.357 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:58.397 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:58.444 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:58.471 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:18:03.195 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:18:03.272 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:05.554 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:05.555 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:18:05.555 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:18:05.555 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:05.555 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:18:05.555 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:18:05.883 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:18:05.883 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:05.884 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:06.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-03-29T11:18:06.745 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:06.904 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:06.906 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:18:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:18:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:18:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:18:06.910 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:06.910 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:06.910 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-29T11:18:06.910 INFO:teuthology.orchestra.run.smithi052.stdout:Install 36 Packages 2024-03-29T11:18:06.910 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:06.910 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 30 M 2024-03-29T11:18:06.911 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 107 M 2024-03-29T11:18:06.911 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:07.720 INFO:teuthology.orchestra.run.smithi052.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-29T11:18:08.220 INFO:teuthology.orchestra.run.smithi052.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-03-29T11:18:08.671 INFO:teuthology.orchestra.run.smithi052.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 4.9 MB/s | 4.7 MB 00:00 2024-03-29T11:18:08.763 INFO:teuthology.orchestra.run.smithi052.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.6 MB/s | 247 kB 00:00 2024-03-29T11:18:08.804 INFO:teuthology.orchestra.run.smithi052.stdout:(5/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-29T11:18:08.988 INFO:teuthology.orchestra.run.smithi052.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-03-29T11:18:09.013 INFO:teuthology.orchestra.run.smithi052.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 792 kB/s | 144 kB 00:00 2024-03-29T11:18:09.030 INFO:teuthology.orchestra.run.smithi052.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-03-29T11:18:09.055 INFO:teuthology.orchestra.run.smithi052.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-29T11:18:09.080 INFO:teuthology.orchestra.run.smithi052.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 768 kB/s | 39 kB 00:00 2024-03-29T11:18:09.114 INFO:teuthology.orchestra.run.smithi052.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T11:18:09.181 INFO:teuthology.orchestra.run.smithi052.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-29T11:18:09.206 INFO:teuthology.orchestra.run.smithi052.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-29T11:18:09.231 INFO:teuthology.orchestra.run.smithi052.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:18:09.257 INFO:teuthology.orchestra.run.smithi052.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:18:09.415 INFO:teuthology.orchestra.run.smithi052.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-29T11:18:09.632 INFO:teuthology.orchestra.run.smithi052.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 12 MB/s | 17 MB 00:01 2024-03-29T11:18:09.766 INFO:teuthology.orchestra.run.smithi052.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 234 kB/s | 125 kB 00:00 2024-03-29T11:18:09.791 INFO:teuthology.orchestra.run.smithi052.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 402 kB/s | 151 kB 00:00 2024-03-29T11:18:09.808 INFO:teuthology.orchestra.run.smithi052.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 248 kB/s | 43 kB 00:00 2024-03-29T11:18:09.884 INFO:teuthology.orchestra.run.smithi052.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-03-29T11:18:09.942 INFO:teuthology.orchestra.run.smithi052.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-29T11:18:09.967 INFO:teuthology.orchestra.run.smithi052.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 855 kB/s | 172 kB 00:00 2024-03-29T11:18:09.992 INFO:teuthology.orchestra.run.smithi052.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-03-29T11:18:10.017 INFO:teuthology.orchestra.run.smithi052.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-03-29T11:18:10.043 INFO:teuthology.orchestra.run.smithi052.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi052.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-29T11:18:10.118 INFO:teuthology.orchestra.run.smithi052.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-29T11:18:10.144 INFO:teuthology.orchestra.run.smithi052.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-29T11:18:10.169 INFO:teuthology.orchestra.run.smithi052.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 627 kB/s | 79 kB 00:00 2024-03-29T11:18:10.194 INFO:teuthology.orchestra.run.smithi052.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-29T11:18:10.236 INFO:teuthology.orchestra.run.smithi052.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 532 kB/s | 36 kB 00:00 2024-03-29T11:18:10.261 INFO:teuthology.orchestra.run.smithi052.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 765 kB/s | 90 kB 00:00 2024-03-29T11:18:10.320 INFO:teuthology.orchestra.run.smithi052.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T11:18:10.345 INFO:teuthology.orchestra.run.smithi052.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-29T11:18:10.370 INFO:teuthology.orchestra.run.smithi052.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-29T11:18:10.375 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:10.375 INFO:teuthology.orchestra.run.smithi052.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-03-29T11:18:10.400 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:10.446 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:10.446 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:12.210 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:12.273 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:18:12.312 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:18:12.342 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:18:12.415 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:18:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:18:12.490 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:18:12.588 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:18:12.673 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:18:12.719 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:18:12.794 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:18:12.845 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:18:12.883 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:18:12.941 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:18:13.005 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:18:13.220 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:18:13.270 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:18:13.308 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:18:13.334 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:18:13.364 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:18:13.443 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:18:13.482 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:13.515 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:18:13.556 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:18:13.600 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:18:13.642 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:18:13.736 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:18:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:18:13.850 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:18:14.000 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:18:14.128 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:14.297 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:18:14.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:14.343 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:15.862 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:15.890 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:15.890 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:15.890 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:18:15.890 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:15.890 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:15.891 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:16.368 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:16.392 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:16.392 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:16.392 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:18:16.392 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:16.392 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:16.392 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:16.613 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:16.636 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:16.636 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:16.636 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:18:16.636 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:16.636 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:16.636 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:16.704 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:18:18.453 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:18:18.454 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:18:18.455 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:18:18.455 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:18:18.455 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:18:18.456 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:18:19.404 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:18:19.405 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:18:19.406 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:19.407 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:19.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-03-29T11:18:20.349 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:20.451 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:20.506 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:20.507 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:18:20.508 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:20.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-03-29T11:18:21.141 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:21.298 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:21.299 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 747 k 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 754 k 2024-03-29T11:18:21.300 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:21.734 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T11:18:21.799 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:21.799 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T11:18:21.800 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:21.800 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:21.800 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:21.800 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:21.800 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:22.717 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:23.007 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:23.007 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:23.007 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:23.008 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:23.008 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:23.008 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:23.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:18:23.813 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:23.970 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:23.970 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 145 k 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 436 k 2024-03-29T11:18:23.971 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:24.306 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 434 kB/s | 145 kB 00:00 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi052.stdout:Total 433 kB/s | 145 kB 00:00 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:24.315 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:24.315 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:24.371 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:24.371 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:24.511 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:24.574 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:24.598 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:24.598 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:24.598 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:18:24.598 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:25.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:25.648 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:25.648 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:25.648 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:25.649 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:25.649 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:25.883 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-03-29T11:18:26.462 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:26.619 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:26.620 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:18:26.620 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:26.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:18:27.251 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:27.408 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:27.409 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-03-29T11:18:27.409 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:27.409 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-03-29T11:18:27.409 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:27.409 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:27.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:27.411 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.9 M 2024-03-29T11:18:27.412 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-03-29T11:18:27.412 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:28.093 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 146 kB/s | 31 kB 00:00 2024-03-29T11:18:28.118 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 64 kB/s | 15 kB 00:00 2024-03-29T11:18:28.251 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T11:18:28.285 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-29T11:18:28.552 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-03-29T11:18:28.602 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-03-29T11:18:28.736 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-03-29T11:18:28.794 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-29T11:18:28.844 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 616 kB/s | 31 kB 00:00 2024-03-29T11:18:28.878 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.3 MB/s | 863 kB 00:00 2024-03-29T11:18:28.903 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-29T11:18:29.045 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.5 MB/s | 1.8 MB 00:01 2024-03-29T11:18:29.049 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:29.049 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.2 MB/s | 6.9 MB 00:01 2024-03-29T11:18:29.068 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:29.086 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:29.086 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:29.255 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:29.256 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:29.927 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:29.997 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:30.032 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:30.079 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:30.141 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:30.246 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:30.372 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:30.463 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:30.630 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:30.666 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:30.690 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:30.935 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:30.964 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:31.727 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:31.728 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:31.728 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:31.728 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:32.083 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:32.084 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:32.085 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:32.085 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:32.085 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:32.085 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:32.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:32.913 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:33.072 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:33.074 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-03-29T11:18:33.075 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:33.076 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 37 M 2024-03-29T11:18:33.076 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2024-03-29T11:18:33.076 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:33.376 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 229 kB/s | 32 kB 00:00 2024-03-29T11:18:33.402 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 720 kB/s | 17 kB 00:00 2024-03-29T11:18:33.427 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-29T11:18:33.594 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2024-03-29T11:18:33.644 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.4 MB/s | 3.0 MB 00:00 2024-03-29T11:18:33.744 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-29T11:18:33.769 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-29T11:18:34.095 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T11:18:34.412 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 617 kB/s | 195 kB 00:00 2024-03-29T11:18:34.570 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-29T11:18:34.587 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T11:18:35.029 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.1 MB/s | 7.4 MB 00:01 2024-03-29T11:18:35.032 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:35.032 INFO:teuthology.orchestra.run.smithi052.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-29T11:18:35.073 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:35.098 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:35.098 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:35.481 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:35.482 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:36.206 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:36.286 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:36.350 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:36.449 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:36.487 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:36.674 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:36.703 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:36.822 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:37.511 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:37.627 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:40.017 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:40.042 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:42.472 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:42.473 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:42.867 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:42.867 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:42.868 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:42.869 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:43.274 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:43.852 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:44.010 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:44.011 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-03-29T11:18:44.012 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:44.013 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2024-03-29T11:18:44.013 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-03-29T11:18:44.013 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:44.176 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 161 kB/s | 18 kB 00:00 2024-03-29T11:18:44.201 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 177 kB/s | 24 kB 00:00 2024-03-29T11:18:44.251 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:18:44.276 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:44.302 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:44.326 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T11:18:44.452 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 134 kB/s | 52 kB 00:00 2024-03-29T11:18:44.577 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-03-29T11:18:44.602 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-29T11:18:44.761 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-03-29T11:18:44.836 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 529 kB/s | 203 kB 00:00 2024-03-29T11:18:44.886 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-29T11:18:44.912 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-29T11:18:44.916 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:44.916 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-29T11:18:44.916 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:44.930 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:44.930 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:45.028 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:45.029 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:45.553 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:45.742 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:45.806 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:45.863 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:45.898 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:45.940 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:46.037 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:46.146 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:46.208 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:46.973 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:47.029 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:47.063 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:47.112 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:47.140 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:47.932 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:47.932 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:47.932 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:47.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:48.409 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:48.410 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:48.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:49.386 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:49.386 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-29T11:18:49.386 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:49.387 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:49.388 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:49.388 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-29T11:18:49.388 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-03-29T11:18:49.388 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:49.388 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.8 M 2024-03-29T11:18:49.388 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-03-29T11:18:49.389 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:49.830 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:18:49.930 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 520 kB/s | 138 kB 00:00 2024-03-29T11:18:50.189 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 223 kB/s | 58 kB 00:00 2024-03-29T11:18:50.214 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T11:18:50.248 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 883 kB/s | 51 kB 00:00 2024-03-29T11:18:50.448 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 5.8 MB 00:00 2024-03-29T11:18:50.451 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:50.451 INFO:teuthology.orchestra.run.smithi052.stdout:Total 6.4 MB/s | 6.8 MB 00:01 2024-03-29T11:18:50.464 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:50.475 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:50.475 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:50.568 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:50.568 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:50.878 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:51.039 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:51.092 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:51.645 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:51.729 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:52.914 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:53.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-03-29T11:18:53.723 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:53.879 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:53.880 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-29T11:18:53.881 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:53.881 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 964 k 2024-03-29T11:18:53.881 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.8 M 2024-03-29T11:18:53.881 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:54.007 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 625 kB/s | 79 kB 00:00 2024-03-29T11:18:54.191 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-03-29T11:18:54.192 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:54.192 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-29T11:18:54.192 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:54.213 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:54.214 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:54.290 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:54.290 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:54.475 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:54.612 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:54.685 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:55.358 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:55.358 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:55.663 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:55.663 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:55.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2024-03-29T11:18:56.548 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:18:56.708 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:18:56.709 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:56.709 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-29T11:18:56.709 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:56.709 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:18:56.709 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 332 k 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.5 M 2024-03-29T11:18:56.710 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:18:56.836 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 547 kB/s | 69 kB 00:00 2024-03-29T11:18:57.528 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 323 kB/s | 264 kB 00:00 2024-03-29T11:18:57.529 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:57.530 INFO:teuthology.orchestra.run.smithi052.stdout:Total 405 kB/s | 332 kB 00:00 2024-03-29T11:18:57.530 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:18:57.540 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:18:57.540 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:18:57.584 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:18:57.584 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:18:57.736 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:18:58.072 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:58.107 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:58.123 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:58.123 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:58.123 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:58.123 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:58.852 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:18:59.208 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:18:59.525 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-03-29T11:19:00.106 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:00.263 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:00.264 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:19:00.265 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:00.265 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:19:00.265 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:00.265 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 127 k 2024-03-29T11:19:00.265 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2024-03-29T11:19:00.265 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:19:00.494 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 557 kB/s | 127 kB 00:00 2024-03-29T11:19:00.494 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:00.494 INFO:teuthology.orchestra.run.smithi052.stdout:Total 553 kB/s | 127 kB 00:00 2024-03-29T11:19:00.494 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:19:00.499 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:19:00.499 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:19:00.559 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:19:00.559 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:19:00.714 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:19:00.819 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:01.248 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:01.542 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:01.542 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:01.542 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:19:01.542 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:01.542 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:01.542 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:01.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-03-29T11:19:02.366 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:02.468 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:02.525 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:02.526 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:02.527 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:02.585 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-03-29T11:19:03.161 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:03.319 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:19:03.320 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 31 k 2024-03-29T11:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 140 k 2024-03-29T11:19:03.321 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:19:03.562 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 129 kB/s | 31 kB 00:00 2024-03-29T11:19:03.567 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:03.567 INFO:teuthology.orchestra.run.smithi052.stdout:Total 128 kB/s | 31 kB 00:00 2024-03-29T11:19:03.567 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:19:03.567 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:19:03.567 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:19:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:19:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:19:03.704 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:19:03.806 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:04.266 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:04.538 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:04.538 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:04.538 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:19:04.538 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:04.538 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:04.538 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:04.835 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-03-29T11:19:05.409 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:05.508 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:05.563 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:05.564 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:05.564 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:05.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-03-29T11:19:06.177 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:06.278 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:06.332 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:06.333 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:06.333 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:06.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-03-29T11:19:06.956 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:07.057 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:07.112 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:07.113 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:07.113 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:07.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-03-29T11:19:07.732 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:07.833 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:07.914 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:07.914 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:07.914 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:07.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-03-29T11:19:08.518 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:08.619 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:08.675 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:08.676 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:08.676 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:08.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-03-29T11:19:09.300 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:09.401 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:09.456 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:09.457 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-29T11:19:09.457 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:09.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-03-29T11:19:10.075 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:10.229 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:10.230 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:10.230 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:10.230 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:10.230 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:19:10.230 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 85 k 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 231 k 2024-03-29T11:19:10.231 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:19:10.431 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 426 kB/s | 85 kB 00:00 2024-03-29T11:19:10.432 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:10.432 INFO:teuthology.orchestra.run.smithi052.stdout:Total 423 kB/s | 85 kB 00:00 2024-03-29T11:19:10.432 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:19:10.440 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:19:10.440 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:19:10.497 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:19:10.498 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:19:10.635 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:19:10.749 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:11.125 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:11.436 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:11.437 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:19:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.437 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:11.437 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:11.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-03-29T11:19:12.247 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:12.407 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:12.407 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.1 M 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 13 M 2024-03-29T11:19:12.408 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:19:13.176 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.0 MB/s | 3.1 MB 00:00 2024-03-29T11:19:13.176 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.0 MB/s | 3.1 MB 00:00 2024-03-29T11:19:13.177 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:19:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:19:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:19:13.263 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:19:13.264 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:19:13.716 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:19:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:13.810 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:13.810 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:19:13.810 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:13.810 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:13.810 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:14.976 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:14.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:14.976 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:19:14.976 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:14.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:14.976 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:15.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-03-29T11:19:15.779 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:16:37 AM UTC. 2024-03-29T11:19:15.934 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:15.935 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-29T11:19:15.936 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-29T11:19:15.936 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-29T11:19:15.936 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:15.936 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 171 k 2024-03-29T11:19:15.936 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 494 k 2024-03-29T11:19:15.936 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-29T11:19:16.278 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 502 kB/s | 171 kB 00:00 2024-03-29T11:19:16.278 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:16.279 INFO:teuthology.orchestra.run.smithi052.stdout:Total 500 kB/s | 171 kB 00:00 2024-03-29T11:19:16.279 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-29T11:19:16.287 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-29T11:19:16.287 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-29T11:19:16.346 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-29T11:19:16.346 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-29T11:19:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-29T11:19:16.621 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:17.334 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:17.334 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:17.335 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-29T11:19:17.335 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:17.335 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:17.335 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-29T11:19:17.590 DEBUG:teuthology.parallel:result is None 2024-03-29T11:19:17.591 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:19:17.808 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:19:17.829 INFO:teuthology.orchestra.run.smithi052.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:17.829 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:17.829 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:19:17.831 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:19:17.970 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:19:17.996 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:17.996 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:17.996 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:19:17.998 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:19:17.998 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:17.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:19:18.027 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:18.027 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:19:18.068 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:19:18.069 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:18.069 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:19:18.096 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:19:18.164 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:18.164 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:19:18.196 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:19:18.267 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:19:18.267 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:18.267 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:19:18.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:19:18.363 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:18.363 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:19:18.393 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:19:18.461 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:19:18.461 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:18.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:19:18.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:19:18.555 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:18.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:19:18.582 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:19:18.649 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:19:18.736 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:19:18.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2024-03-29T11:19:18.739 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2024-03-29T11:19:18.767 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:19:18.767 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:19:18.807 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:19:18.839 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:19:18.839 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:18.874 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:18.917 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:19:18.917 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:19:18.917 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622695', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbbdhHQxBQkJRN769eMBmzh/0mCvr4Sar+m7eLmjdJTrB7PzcV4p69K2BYniUODFRvANxG2lols4JsHnC2ZZ3Q=', 'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNA5jl3+PWU8sr0KWl1Dye4j9g+m0x3sFQfImSX++M/iQdOsHG4erVwWf4OzwDEZKro2PkTqEYhDkrEndVGrY+Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:19:18.917 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:18.942 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:18.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:18.997 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:19.031 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:19.031 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:19.077 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:19.077 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:19.133 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-29 11:19:14.104725824 +0000 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-29 11:14:28.671393337 +0000 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-29 11:14:28.671393337 +0000 2024-03-29T11:19:19.134 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-29 11:14:28.671393337 +0000 2024-03-29T11:19:19.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:19.198 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-29T11:19:19.198 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-29T11:19:19.198 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.5849e-05 s, 5.3 MB/s 2024-03-29T11:19:19.199 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:19.255 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-29 11:19:14.104725824 +0000 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-29 11:14:29.004388659 +0000 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-29 11:14:29.004388659 +0000 2024-03-29T11:19:19.310 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-29 11:14:29.004388659 +0000 2024-03-29T11:19:19.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:19.375 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-29T11:19:19.375 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-29T11:19:19.375 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.204e-05 s, 6.2 MB/s 2024-03-29T11:19:19.376 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:19.432 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-29 11:19:14.104725824 +0000 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-29 11:14:29.340383938 +0000 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-29 11:14:29.340383938 +0000 2024-03-29T11:19:19.486 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-29 11:14:29.340383938 +0000 2024-03-29T11:19:19.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-29T11:19:19.552 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-29T11:19:19.552 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.9259e-05 s, 5.7 MB/s 2024-03-29T11:19:19.553 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:19.608 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-29 11:19:14.105725810 +0000 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-29 11:14:29.681379147 +0000 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-29 11:14:29.681379147 +0000 2024-03-29T11:19:19.664 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-29 11:14:29.681379147 +0000 2024-03-29T11:19:19.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:19.729 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-29T11:19:19.729 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-29T11:19:19.729 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.3752e-05 s, 5.5 MB/s 2024-03-29T11:19:19.730 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:19.786 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:19.786 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:19.786 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:19.802 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:19.802 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:19.857 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:19.857 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:19.857 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:19:19.857 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:19.858 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:19.858 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 11:17:10.195207867 +0000 2024-03-29T11:19:19.858 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 11:14:18.428084929 +0000 2024-03-29T11:19:19.858 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 11:14:18.428084929 +0000 2024-03-29T11:19:19.858 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 11:14:18.428084929 +0000 2024-03-29T11:19:19.858 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:19.923 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T11:19:19.923 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T11:19:19.923 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000270149 s, 1.9 MB/s 2024-03-29T11:19:19.924 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:19.980 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 11:17:10.196207844 +0000 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 11:14:18.791076554 +0000 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 11:14:18.791076554 +0000 2024-03-29T11:19:20.034 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 11:14:18.791076554 +0000 2024-03-29T11:19:20.035 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:20.100 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T11:19:20.100 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T11:19:20.100 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000198706 s, 2.6 MB/s 2024-03-29T11:19:20.101 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:20.158 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 11:17:10.196207844 +0000 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 11:14:19.140068503 +0000 2024-03-29T11:19:20.213 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 11:14:19.140068503 +0000 2024-03-29T11:19:20.214 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 11:14:19.140068503 +0000 2024-03-29T11:19:20.214 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:20.278 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T11:19:20.278 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T11:19:20.279 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000227007 s, 2.3 MB/s 2024-03-29T11:19:20.280 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:20.335 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 11:17:10.196207844 +0000 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 11:14:19.485060544 +0000 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 11:14:19.485060544 +0000 2024-03-29T11:19:20.390 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 11:14:19.485060544 +0000 2024-03-29T11:19:20.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:20.456 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T11:19:20.456 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T11:19:20.456 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000191794 s, 2.7 MB/s 2024-03-29T11:19:20.457 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:20.512 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:20.512 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:20.513 INFO:tasks.ceph:Generating config... 2024-03-29T11:19:20.514 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:19:20.514 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:19:20.514 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:19:20.514 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:19:20.514 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:19:20.514 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:19:20.514 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:19:20.514 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:19:20.514 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:19:20.514 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:19:20.514 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:19:20.514 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:19:20.515 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:19:20.515 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:19:20.515 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:19:20.515 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:19:20.515 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:19:20.515 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:19:20.516 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:19:20.517 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:19:20.517 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:19:20.517 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:19:20.517 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:19:20.517 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:19:20.517 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:19:20.517 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:19:20.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:20.572 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:19:20.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:19:20.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:20.650 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.52:6789]'), ('mon.c', '[v1:172.21.15.52:6790]'), ('mon.b', '[v1:172.21.15.79:6789]')] 2024-03-29T11:19:20.650 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.52:6789],[v1:172.21.15.52:6790],[v1:172.21.15.79:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:19:20.651 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:20.651 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:20.704 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.52:6789]' --addv c '[v1:172.21.15.52:6790]' --addv b '[v1:172.21.15.79:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 17 (quincy) 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:election_strategy: 1 2024-03-29T11:19:20.790 INFO:teuthology.orchestra.run.smithi052.stdout:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:20.791 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:19:20.792 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:20.806 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3b9c51c5-3002-4ea5-be53-d626ccc5bde4... 2024-03-29T11:19:20.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:20.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:20.896 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = "[v1:172.21.15.52:6789],[v1:172.21.15.52:6790],[v1:172.21.15.79:6789]" 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: lockdep = True 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: mon client directed command retry = 5 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: mon election default strategy = 1 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: ms bind msgr2 = False 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: ms inject socket failures = 5000 2024-03-29T11:19:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: ms type = async 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2024-03-29T11:19:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug trim objects = true 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore zero block detection = True 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 20 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 20 2024-03-29T11:19:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 10 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: osd class default list = * 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: osd class load list = * 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: osd client watch timeout = 120 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify missing on start = True 2024-03-29T11:19:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue = debug_random 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2024-03-29T11:19:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: mon scrub interval = 300 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on pool no app = False 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: paxos service trim min = 10 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: debug objecter = 20 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: debug rados = 20 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.c] 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.b] 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-03-29T11:19:20.909 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.910 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2024-03-29T11:19:20.911 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2024-03-29T11:19:20.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = "[v1:172.21.15.52:6789],[v1:172.21.15.52:6790],[v1:172.21.15.79:6789]" 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: lockdep = True 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: mon client directed command retry = 5 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: mon election default strategy = 1 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: ms bind msgr2 = False 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject socket failures = 5000 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: ms type = async 2024-03-29T11:19:20.913 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 600 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2024-03-29T11:19:20.914 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug trim objects = true 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.915 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore zero block detection = True 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 20 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 20 2024-03-29T11:19:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 10 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: osd class default list = * 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: osd class load list = * 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: osd client watch timeout = 120 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify missing on start = True 2024-03-29T11:19:20.917 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue = debug_random 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:20.919 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:20.920 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon scrub interval = 300 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on pool no app = False 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: paxos service trim min = 10 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T11:19:20.921 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: debug objecter = 20 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout: debug rados = 20 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.c] 2024-03-29T11:19:20.922 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.b] 2024-03-29T11:19:20.924 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:19:20.924 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:19:20.979 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:19:20.979 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:20.979 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:19:20.993 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:20.993 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:19:21.047 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:19:21.048 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:21.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:21.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:21.126 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:21.126 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:21.180 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:19:21.181 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:21.181 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:21.181 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:21.220 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:21.220 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:21.275 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:19:21.275 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:19:21.275 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:21.330 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:21.333 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:21.399 INFO:teuthology.orchestra.run.smithi079.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:21.401 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:19:21.401 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:19:21.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:21.443 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:21.459 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:21.501 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:21.516 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:19:21.516 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:21.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:21.542 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:21.542 INFO:tasks.ceph:role: osd.0 2024-03-29T11:19:21.542 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:19:21.543 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:21.610 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:21.610 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:21.611 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:21.611 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:21.611 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:21.643 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-29T11:19:21.645 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-29T11:19:21.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:21.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:21.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:21.853 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:21.853 INFO:tasks.ceph:role: osd.1 2024-03-29T11:19:21.853 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:19:21.854 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:21.919 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:21.920 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:21.952 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-29T11:19:21.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-29T11:19:21.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:22.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:22.105 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:22.173 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:22.173 INFO:tasks.ceph:role: osd.2 2024-03-29T11:19:22.173 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:19:22.173 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:22.239 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:22.239 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:22.240 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:22.240 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:22.240 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:22.271 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-29T11:19:22.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-29T11:19:22.273 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:22.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:22.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:22.480 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:22.480 INFO:tasks.ceph:role: osd.3 2024-03-29T11:19:22.480 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:19:22.480 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:22.578 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-29T11:19:22.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-29T11:19:22.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:22.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:22.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:22.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:22.851+0000 7ffaac868640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:19:22.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:22.851+0000 7ffaac868640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:19:22.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:22.851+0000 7ffaac868640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:22.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:22.851+0000 7ffaac868640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:22.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:22.851+0000 7ffaac868640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:22.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:22.852+0000 7ffaac868640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:19:25.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:25.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:25.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:25.815+0000 7f00d1f1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:19:25.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:25.815+0000 7f00d1f1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:19:25.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:25.815+0000 7f00d1f1b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:25.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:25.815+0000 7f00d1f1b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:25.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:25.815+0000 7f00d1f1b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:25.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:25.816+0000 7f00d1f1b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:19:28.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:28.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:28.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:28.785+0000 7f87bd419640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:19:28.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:28.785+0000 7f87bd419640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:19:28.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:28.785+0000 7f87bd419640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:28.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:28.785+0000 7f87bd419640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:28.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:28.785+0000 7f87bd419640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:28.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:28.785+0000 7f87bd419640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:19:31.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:31.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:31.755+0000 7fb6413df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:19:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:31.755+0000 7fb6413df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:19:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:31.755+0000 7fb6413df640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:31.755+0000 7fb6413df640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:31.755+0000 7fb6413df640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:31.755+0000 7fb6413df640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:19:34.588 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:34.619 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:34.651 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:34.651 INFO:tasks.ceph:role: osd.4 2024-03-29T11:19:34.651 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:19:34.651 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:34.729 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:35.236 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T11:19:35.238 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T11:19:35.238 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:35.375 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:35.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:35.481 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:35.481 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:35.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:19:35.481 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:35.556 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:35.557 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:36.013 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T11:19:36.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T11:19:36.016 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:36.113 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:36.146 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:36.214 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:36.214 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:36.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:19:36.215 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:36.289 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:36.290 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:36.782 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T11:19:36.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T11:19:36.784 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:36.942 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:36.977 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:37.045 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:37.045 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:37.045 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:19:37.045 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:37.120 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:37.120 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:37.121 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:37.606 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T11:19:37.608 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T11:19:37.608 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:37.746 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:37.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:37.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:37.907+0000 7f347a01b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:37.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:37.907+0000 7f347a01b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:37.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:37.907+0000 7f347a01b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:37.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:37.907+0000 7f347a01b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:37.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:37.907+0000 7f347a01b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:37.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:37.908+0000 7f347a01b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:40.724 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:40.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:40.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:40.866+0000 7fba869a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:40.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:40.866+0000 7fba869a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:40.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:40.866+0000 7fba869a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:40.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:40.866+0000 7fba869a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:40.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:40.866+0000 7fba869a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:40.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:40.866+0000 7fba869a8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:43.697 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:43.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:43.843+0000 7f6bd8212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:43.843+0000 7f6bd8212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:43.843+0000 7f6bd8212640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:43.843+0000 7f6bd8212640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:43.843+0000 7f6bd8212640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:43.844+0000 7f6bd8212640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:46.672 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:46.704 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:46.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:46.820+0000 7fa219e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:46.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:46.820+0000 7fa219e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:46.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:46.820+0000 7fa219e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:46.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:46.820+0000 7fa219e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:46.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:46.820+0000 7fa219e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:46.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:46.821+0000 7fa219e15640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:49.656 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:49.687 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:49.688 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:49.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:49.719 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:49.720 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:49.790 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:49.790 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:49.860 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:49.860 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:49.930 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:49.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:49.999 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:49.999 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:50.030 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:50.030 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:50.100 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:50.100 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:50.170 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:50.170 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:50.240 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:50.240 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:50.309 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:50.310 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:50.327 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:50.327 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:50.365 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:50.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:50.369 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:50.408 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.y] 2024-03-29T11:19:50.408 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC5owZmTI8kFRAADQdS2WsEa+vrCbFLuCodxQ== 2024-03-29T11:19:50.408 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2024-03-29T11:19:50.408 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC6owZm4IHQMhAA3ZgBwCWnngityNk6dkn2VA== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC9owZmQMqqMBAAxOP6ux4tnZmBA5cpPkQopA== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDAowZmpi7bLhAAZU4ntqcIC4xFOtIonbWsgQ== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.3] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDDowZmoksRLRAAxcX9lgXlymVZyIpTenFleg== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC5owZmCIw/GRAAvq4OGI5iTQorLKyvQO4Crw== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.4] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDJowZmC4kzNhAAHgiReQ/iuz6NNqLEDMlQiA== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.5] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDMowZmd6qrMxAAscpsgCJhfUG0kiqee9y0lg== 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.6] 2024-03-29T11:19:50.409 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDPowZm545OMhAAsK0AknuIgokhg0Siw0h4Tw== 2024-03-29T11:19:50.410 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.7] 2024-03-29T11:19:50.410 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDSowZmAo3zMBAAkaGSEmmlpZOwvctTn0WNDg== 2024-03-29T11:19:50.410 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-03-29T11:19:50.410 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC5owZmz2LqGxAA0RfoGlAtW9gsajYeADqV9Q== 2024-03-29T11:19:50.410 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2024-03-29T11:19:50.410 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC5owZmtBhTHxAAoT77DICsdNmHZh+aamI5kg== 2024-03-29T11:19:50.433 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.y] 2024-03-29T11:19:50.433 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQC5owZmTI8kFRAADQdS2WsEa+vrCbFLuCodxQ== 2024-03-29T11:19:50.433 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-03-29T11:19:50.433 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQC6owZm4IHQMhAA3ZgBwCWnngityNk6dkn2VA== 2024-03-29T11:19:50.433 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQC9owZmQMqqMBAAxOP6ux4tnZmBA5cpPkQopA== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDAowZmpi7bLhAAZU4ntqcIC4xFOtIonbWsgQ== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDDowZmoksRLRAAxcX9lgXlymVZyIpTenFleg== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQC5owZmCIw/GRAAvq4OGI5iTQorLKyvQO4Crw== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.4] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDJowZmC4kzNhAAHgiReQ/iuz6NNqLEDMlQiA== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.5] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDMowZmd6qrMxAAscpsgCJhfUG0kiqee9y0lg== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.6] 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDPowZm545OMhAAsK0AknuIgokhg0Siw0h4Tw== 2024-03-29T11:19:50.434 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.7] 2024-03-29T11:19:50.435 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDSowZmAo3zMBAAkaGSEmmlpZOwvctTn0WNDg== 2024-03-29T11:19:50.435 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-03-29T11:19:50.435 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQC5owZmz2LqGxAA0RfoGlAtW9gsajYeADqV9Q== 2024-03-29T11:19:50.435 INFO:teuthology.orchestra.run.smithi079.stdout:[client.1] 2024-03-29T11:19:50.435 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQC5owZmtBhTHxAAoT77DICsdNmHZh+aamI5kg== 2024-03-29T11:19:50.436 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:50.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:50.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.534 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.587 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.590 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.642 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.697 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:50.751 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:50.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.805 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.856 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.912 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:50.966 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:51.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:51.020 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:51.074 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:51.076 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:51.130 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:51.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:51.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:51.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:51.280 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:51.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:51.443 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:51.470 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:51.498 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:51.594 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:51.621 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:51.623 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:51.676 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:51.676 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:51.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:51.678 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:51.678 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:51.678 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:51.680 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:51.680 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:51.680 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:51.718 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:51.719 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:51.719 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:51.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:51.721 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:51.721 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:51.721 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:51.723 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:51.723 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:51.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:51.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.849+0000 7f606384e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:51.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.849+0000 7f606384e640 10 monclient: build_initial_monmap 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.849+0000 7f606384e640 10 monclient: monmap: 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:51.850251+0000 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:51.850251+0000 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:51.826 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.849+0000 7f606384e640 1 Processor -- start 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 1 -- start start 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 10 monclient: init 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 10 monclient(hunting): picked mon.noname-b con 0x7f605c0b8240 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 10 monclient(hunting): picked mon.noname-a con 0x7f605c0b1f60 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.850+0000 7f606384e640 10 monclient(hunting): picked mon.noname-c con 0x7f605c09c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f606384e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c09bd80 con 0x7f605c0b1f60 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f606384e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c09b960 con 0x7f605c09c8c0 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f606384e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c0b2ec0 con 0x7f605c0b8240 2024-03-29T11:19:51.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f6060de9640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f605c0b1f60 0x7f605c09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:52238/0 (socket says 172.21.15.52:52238) 2024-03-29T11:19:51.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f6060de9640 1 -- 172.21.15.52:0/4009045932 learned_addr learned my addr 172.21.15.52:0/4009045932 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:19:51.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f606384e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:51.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:51.851+0000 7f606384e640 10 monclient(hunting): authenticate will time out at 921.991227s 2024-03-29T11:19:54.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): tick 2024-03-29T11:19:54.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:54.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 -- 172.21.15.52:0/4009045932 >> v1:172.21.15.52:6790/0 conn(0x7f605c0b8240 legacy=0x7f605c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 -- 172.21.15.52:0/4009045932 >> v1:172.21.15.79:6789/0 conn(0x7f605c09c8c0 legacy=0x7f605c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 -- 172.21.15.52:0/4009045932 >> v1:172.21.15.52:6789/0 conn(0x7f605c0b1f60 legacy=0x7f605c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f60400036e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f60400072d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f605c09c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 -- 172.21.15.52:0/4009045932 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c13fb60 con 0x7f60400036e0 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 -- 172.21.15.52:0/4009045932 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604000f8d0 con 0x7f605c09c8c0 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 1 -- 172.21.15.52:0/4009045932 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6040010b10 con 0x7f60400072d0 2024-03-29T11:19:54.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:54.851+0000 7f6052ffd640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734032627 0 0) 0x7f605c13fb60 con 0x7f60400036e0 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6048003520 con 0x7f60400036e0 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278497632 0 0) 0x7f6040010b10 con 0x7f60400072d0 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:56.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.016+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605c13fb60 con 0x7f60400072d0 2024-03-29T11:19:57.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572057824 0 0) 0x7f605c13fb60 con 0x7f60400072d0 2024-03-29T11:19:57.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 >> v1:172.21.15.79:6789/0 conn(0x7f605c09c8c0 legacy=0x7f604000d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 >> v1:172.21.15.52:6789/0 conn(0x7f60400036e0 legacy=0x7f6040005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 monclient: found mon.noname-b 2024-03-29T11:19:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:19:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6040011d70 con 0x7f60400072d0 2024-03-29T11:19:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f604c0015c0 con 0x7f60400072d0 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f606384e640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f606384e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: dump: 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.007 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f606384e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f604c001790 con 0x7f60400072d0 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 1 -- 172.21.15.52:0/4009045932 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f604c001b20 con 0x7f60400072d0 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f6053fff640 10 monclient: dump: 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.009 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f606384e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.017+0000 7f606384e640 4 set_mon_vals no callback set 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 10 monclient: shutdown 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 1 -- 172.21.15.52:0/4009045932 >> v1:172.21.15.52:6790/0 conn(0x7f60400072d0 legacy=0x7f6040009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 1 -- 172.21.15.52:0/4009045932 shutdown_connections 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 1 -- 172.21.15.52:0/4009045932 >> 172.21.15.52:0/4009045932 conn(0x7f605c066b00 msgr2=0x7f605c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 1 -- 172.21.15.52:0/4009045932 shutdown_connections 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.018+0000 7f606384e640 1 -- 172.21.15.52:0/4009045932 wait complete. 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.019+0000 7f606384e640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.019+0000 7f606384e640 10 monclient: monmap: 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.011 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.019+0000 7f606384e640 1 librados: starting msgr at 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.019+0000 7f606384e640 1 librados: starting objecter 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.019+0000 7f606384e640 1 Processor -- start 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 1 -- start start 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 1 librados: setting wanted keys 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 1 librados: calling monclient init 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient: init 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient(hunting): picked mon.noname-a con 0x7f60400036e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:57.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient(hunting): picked mon.noname-c con 0x7f60400072d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient(hunting): picked mon.noname-b con 0x7f605c09c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c1c8890 con 0x7f60400036e0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c1c8b70 con 0x7f605c09c8c0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c1db360 con 0x7f60400072d0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f606384e640 10 monclient(hunting): authenticate will time out at 927.160966s 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f60615ea640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f60400036e0 0x7f605c1dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33368/0 (socket says 172.21.15.52:33368) 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f6060de9640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f60400072d0 0x7f605c139db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58898/0 (socket says 172.21.15.52:58898) 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.020+0000 7f60615ea640 1 -- 172.21.15.52:0/3933951084 learned_addr learned my addr 172.21.15.52:0/3933951084 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.021+0000 7f6051ffb640 10 client.?.objecter ms_handle_connect 0x7f60400036e0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.021+0000 7f6051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.021+0000 7f6051ffb640 10 client.?.objecter ms_handle_connect 0x7f605c09c8c0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.021+0000 7f6051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.021+0000 7f6051ffb640 10 client.?.objecter ms_handle_connect 0x7f60400072d0 2024-03-29T11:19:57.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.021+0000 7f6051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.022+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141172457 0 0) 0x7f605c1db360 con 0x7f60400072d0 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.022+0000 7f6051ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.022+0000 7f6051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.022+0000 7f6051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.022+0000 7f6051ffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6040010330 con 0x7f60400072d0 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199167027 0 0) 0x7f605c1c8890 con 0x7f60400036e0 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605c1db360 con 0x7f60400036e0 2024-03-29T11:19:57.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426663894 0 0) 0x7f605c1db360 con 0x7f60400036e0 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f605c1c8890 con 0x7f60400036e0 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f60580018b0 con 0x7f60400036e0 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.015 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 785398305 0 0) 0x7f605c1c8890 con 0x7f60400036e0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 >> v1:172.21.15.52:6790/0 conn(0x7f60400072d0 legacy=0x7f605c139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 >> v1:172.21.15.79:6789/0 conn(0x7f605c09c8c0 legacy=0x7f605c0b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 monclient: found mon.a 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c1db4f0 con 0x7f60400036e0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f6051ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f606384e640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f606384e640 10 monclient: _renew_subs 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f606384e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:57.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.023+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c1db760 con 0x7f60400036e0 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60580035f0 con 0x7f60400036e0 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f60580037a0 con 0x7f60400036e0 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 10 monclient: dump: 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.017 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6058003d10 con 0x7f60400036e0 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f606384e640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f60537fe640 4 set_mon_vals no callback set 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f606384e640 10 monclient: _renew_subs 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f606384e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c13a6a0 con 0x7f60400036e0 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f606384e640 1 librados: init done 2024-03-29T11:19:57.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 34185980 0 0) 0x7f6058005780 con 0x7f60400036e0 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 10 client.4113.objecter ms_dispatch 0x7f605c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.024+0000 7f6051ffb640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.025+0000 7f606384e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.025+0000 7f606384e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.025+0000 7f606384e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.025+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c13a950 con 0x7f60400036e0 2024-03-29T11:19:57.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.029+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6058005940 con 0x7f60400036e0 2024-03-29T11:19:57.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.029+0000 7f6051ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.029+0000 7f6051ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.103+0000 7f606384e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:57.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.103+0000 7f606384e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:57.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.103+0000 7f606384e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:57.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.103+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f605c0b0810 con 0x7f60400036e0 2024-03-29T11:19:57.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.105+0000 7f6051ffb640 1 -- 172.21.15.52:0/3933951084 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6058005ff0 con 0x7f60400036e0 2024-03-29T11:19:57.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.105+0000 7f6051ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:57.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.105+0000 7f6051ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:57.082 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-03-29T11:19:57.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 10 librados: watch_flush enter 2024-03-29T11:19:57.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 10 librados: watch_flush exit 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 10 monclient: shutdown 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 >> v1:172.21.15.52:6789/0 conn(0x7f60400036e0 legacy=0x7f605c1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 shutdown_connections 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 >> 172.21.15.52:0/3933951084 conn(0x7f605c066b00 msgr2=0x7f605c0b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 shutdown_connections 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 1 -- 172.21.15.52:0/3933951084 wait complete. 2024-03-29T11:19:57.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:57.106+0000 7f606384e640 1 librados: shutdown 2024-03-29T11:19:57.095 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:57.095 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:57.095 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:57.096 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:57.125 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:57.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:57.191 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:57.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:57.259 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-29T11:19:57.259 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:57.325 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:57.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:57.352 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:57.352 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:57.420 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:57.420 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:57.476 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:57.500+0000 7fd29afb6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:57.488 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T11:19:57.488 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:57.507 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:57.531+0000 7f250a041040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:57.533 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:57.556+0000 7fd29afb6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:57.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new ae9f13d7-d829-45b6-99ab-f3024ff4f4fb 0 2024-03-29T11:19:57.563 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:57.587+0000 7f250a041040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.719+0000 7f40c6d46640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.719+0000 7f40c6d46640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.719+0000 7f40c6d46640 10 monclient: monmap: 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:57.719980+0000 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:57.719980+0000 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:57.707 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.719+0000 7f40c6d46640 1 Processor -- start 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.719+0000 7f40c6d46640 1 -- start start 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.719+0000 7f40c6d46640 10 monclient: init 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient(hunting): picked mon.noname-a con 0x7f40c009ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient(hunting): picked mon.noname-b con 0x7f40c009c870 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient(hunting): picked mon.noname-c con 0x7f40c00b4360 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c009dec0 con 0x7f40c009ac40 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c009da30 con 0x7f40c00b4360 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c00b39c0 con 0x7f40c009c870 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c6d46640 10 monclient(hunting): authenticate will time out at 923.890595s 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c52e3640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f40c00b4360 0x7f40c014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55730/0 (socket says 172.21.15.79:55730) 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40c52e3640 1 -- 172.21.15.79:0/2123012660 learned_addr learned my addr 172.21.15.79:0/2123012660 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538226957 0 0) 0x7f40c009da30 con 0x7f40c00b4360 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b0003520 con 0x7f40c00b4360 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421330600 0 0) 0x7f40c00b39c0 con 0x7f40c009c870 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.720+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c009da30 con 0x7f40c009c870 2024-03-29T11:19:57.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217646331 0 0) 0x7f40b4001110 con 0x7f40c009ac40 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c00b39c0 con 0x7f40c009ac40 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3086697394 0 0) 0x7f40b0003520 con 0x7f40c00b4360 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 >> v1:172.21.15.52:6790/0 conn(0x7f40c009c870 legacy=0x7f40c013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 >> v1:172.21.15.52:6789/0 conn(0x7f40c009ac40 legacy=0x7f40c00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c00b5160 con 0x7f40c00b4360 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f40b8001a10 con 0x7f40c00b4360 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: dump: 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.711 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 5 monclient: authenticate success, global_id 4099 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40b8001bf0 con 0x7f40c00b4360 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 1 -- 172.21.15.79:0/2123012660 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f40b8002000 con 0x7f40c00b4360 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 4 set_mon_vals no callback set 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40bf7fe640 10 monclient: dump: 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.712 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 10 monclient: shutdown 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 1 -- 172.21.15.79:0/2123012660 >> v1:172.21.15.79:6789/0 conn(0x7f40c00b4360 legacy=0x7f40c014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 1 -- 172.21.15.79:0/2123012660 shutdown_connections 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 1 -- 172.21.15.79:0/2123012660 >> 172.21.15.79:0/2123012660 conn(0x7f40c0066b00 msgr2=0x7f40c0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 1 -- 172.21.15.79:0/2123012660 shutdown_connections 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.721+0000 7f40c6d46640 1 -- 172.21.15.79:0/2123012660 wait complete. 2024-03-29T11:19:57.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 10 monclient: monmap: 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 1 librados: starting msgr at 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 1 librados: starting objecter 2024-03-29T11:19:57.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 1 Processor -- start 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 1 -- start start 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 1 librados: setting wanted keys 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 1 librados: calling monclient init 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.722+0000 7f40c6d46640 10 monclient: init 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient(hunting): picked mon.noname-b con 0x7f40c009ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient(hunting): picked mon.noname-c con 0x7f40c00aee50 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient(hunting): picked mon.noname-a con 0x7f40c00ae810 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c01d7b20 con 0x7f40c00ae810 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c00b00d0 con 0x7f40c009ac40 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c00ada70 con 0x7f40c00aee50 2024-03-29T11:19:57.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c6d46640 10 monclient(hunting): authenticate will time out at 923.893570s 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c4ae2640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f40c009ac40 0x7f40c01d7fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55732/0 (socket says 172.21.15.79:55732) 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40c4ae2640 1 -- 172.21.15.79:0/2707889228 learned_addr learned my addr 172.21.15.79:0/2707889228 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 client.?.objecter ms_handle_connect 0x7f40c009ac40 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 client.?.objecter ms_handle_connect 0x7f40c00ae810 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268834268 0 0) 0x7f40c00b00d0 con 0x7f40c009ac40 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a4004eb0 con 0x7f40c009ac40 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 client.?.objecter ms_handle_connect 0x7f40c00aee50 2024-03-29T11:19:57.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807015358 0 0) 0x7f40c01d7b20 con 0x7f40c00ae810 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c00b00d0 con 0x7f40c00ae810 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491747553 0 0) 0x7f40c00ada70 con 0x7f40c00aee50 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c01d7b20 con 0x7f40c00aee50 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 701982508 0 0) 0x7f40a4004eb0 con 0x7f40c009ac40 2024-03-29T11:19:57.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c00ada70 con 0x7f40c009ac40 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.723+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f40b4001860 con 0x7f40c009ac40 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.718 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3254309540 0 0) 0x7f40c01d7b20 con 0x7f40c00aee50 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40a4004eb0 con 0x7f40c00aee50 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f40ac001750 con 0x7f40c00aee50 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.719 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549949649 0 0) 0x7f40c00b00d0 con 0x7f40c00ae810 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c01d7b20 con 0x7f40c00ae810 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879373544 0 0) 0x7f40c00ada70 con 0x7f40c009ac40 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 >> v1:172.21.15.52:6790/0 conn(0x7f40c00aee50 legacy=0x7f40c00b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 >> v1:172.21.15.52:6789/0 conn(0x7f40c00ae810 legacy=0x7f40c00b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 monclient: found mon.b 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c01d7640 con 0x7f40c009ac40 2024-03-29T11:19:57.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 5 monclient: authenticate success, global_id 4102 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 10 monclient: _renew_subs 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c00b17b0 con 0x7f40c009ac40 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 10 monclient: _renew_subs 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c00b1b50 con 0x7f40c009ac40 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40c6d46640 1 librados: init done 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40b4001d50 con 0x7f40c009ac40 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f40b40037b0 con 0x7f40c009ac40 2024-03-29T11:19:57.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 monclient: dump: 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40beffd640 4 set_mon_vals no callback set 2024-03-29T11:19:57.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f40b4003da0 con 0x7f40c009ac40 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1672340842 0 0) 0x7f40b40064f0 con 0x7f40c009ac40 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 10 client.4102.objecter ms_dispatch 0x7f40c0000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 3 client.4102.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 3 client.4102.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.724+0000 7f40bd7fa640 20 client.4102.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.725+0000 7f40c6d46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.725+0000 7f40c6d46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.725+0000 7f40c6d46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.725+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a0005080 con 0x7f40c009ac40 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.727+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f40b4004390 con 0x7f40c009ac40 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.727+0000 7f40bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.727+0000 7f40bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.818+0000 7f40c6d46640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae9f13d7-d829-45b6-99ab-f3024ff4f4fb", "id": 0}] 2024-03-29T11:19:57.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.818+0000 7f40c6d46640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae9f13d7-d829-45b6-99ab-f3024ff4f4fb", "id": 0}] 2024-03-29T11:19:57.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.818+0000 7f40c6d46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.818+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ae9f13d7-d829-45b6-99ab-f3024ff4f4fb", "id": 0} v 0) v1 -- 0x7f40a00058b0 con 0x7f40c009ac40 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.822+0000 7f40bd7fa640 1 -- 172.21.15.79:0/2707889228 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae9f13d7-d829-45b6-99ab-f3024ff4f4fb", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 822658015 0 732707515) 0x7f40b4005890 con 0x7f40c009ac40 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.822+0000 7f40bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae9f13d7-d829-45b6-99ab-f3024ff4f4fb", "id": 0}] 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.822+0000 7f40bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.798 INFO:teuthology.orchestra.run.smithi079.stdout:0 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.822+0000 7f40c6d46640 10 librados: watch_flush enter 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 10 librados: watch_flush exit 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 20 client.4102.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 10 client.4102.objecter successfully canceled tick 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 10 monclient: shutdown 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 >> v1:172.21.15.79:6789/0 conn(0x7f40c009ac40 legacy=0x7f40c01d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 shutdown_connections 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 >> 172.21.15.79:0/2707889228 conn(0x7f40c0066b00 msgr2=0x7f40c013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 shutdown_connections 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 1 -- 172.21.15.79:0/2707889228 wait complete. 2024-03-29T11:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.823+0000 7f40c6d46640 1 librados: shutdown 2024-03-29T11:19:57.809 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new b7490542-1dbf-4ecd-96e0-1008fc1c34a2 1 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.932+0000 7f574103e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.932+0000 7f574103e640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.932+0000 7f574103e640 10 monclient: monmap: 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:57.933422+0000 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:57.933422+0000 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:57.909 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:57.910 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 1 Processor -- start 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 1 -- start start 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient: init 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient(hunting): picked mon.noname-c con 0x7f573c09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient(hunting): picked mon.noname-b con 0x7f573c09c870 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient(hunting): picked mon.noname-a con 0x7f573c0b4360 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c09dec0 con 0x7f573c0b4360 2024-03-29T11:19:57.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c09da30 con 0x7f573c09ac40 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0b39c0 con 0x7f573c09c870 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f574103e640 10 monclient(hunting): authenticate will time out at 924.103863s 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f573ad9d640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f573c09ac40 0x7f573c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55744/0 (socket says 172.21.15.79:55744) 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.933+0000 7f573ad9d640 1 -- 172.21.15.79:0/3448718353 learned_addr learned my addr 172.21.15.79:0/3448718353 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543620452 0 0) 0x7f573c09da30 con 0x7f573c09ac40 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5724003520 con 0x7f573c09ac40 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240787393 0 0) 0x7f5734001490 con 0x7f573c0b4360 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c09da30 con 0x7f573c0b4360 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420954994 0 0) 0x7f573c0b39c0 con 0x7f573c09c870 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5734001490 con 0x7f573c09c870 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1257279891 0 0) 0x7f5724003520 con 0x7f573c09ac40 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 >> v1:172.21.15.52:6790/0 conn(0x7f573c09c870 legacy=0x7f573c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 >> v1:172.21.15.52:6789/0 conn(0x7f573c0b4360 legacy=0x7f573c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0b5160 con 0x7f573c09ac40 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f572c001690 con 0x7f573c09ac40 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: dump: 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.921 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f572c001870 con 0x7f573c09ac40 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 1 -- 172.21.15.79:0/3448718353 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f572c001c80 con 0x7f573c09ac40 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 4 set_mon_vals no callback set 2024-03-29T11:19:57.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f5739d9b640 10 monclient: dump: 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 10 monclient: shutdown 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 1 -- 172.21.15.79:0/3448718353 >> v1:172.21.15.79:6789/0 conn(0x7f573c09ac40 legacy=0x7f573c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.934+0000 7f574103e640 1 -- 172.21.15.79:0/3448718353 shutdown_connections 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 1 -- 172.21.15.79:0/3448718353 >> 172.21.15.79:0/3448718353 conn(0x7f573c066b00 msgr2=0x7f573c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 1 -- 172.21.15.79:0/3448718353 shutdown_connections 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 1 -- 172.21.15.79:0/3448718353 wait complete. 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 10 monclient: monmap: 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:57.924 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 1 librados: starting msgr at 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.935+0000 7f574103e640 1 librados: starting objecter 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 Processor -- start 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 -- start start 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 librados: setting wanted keys 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 librados: calling monclient init 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient: init 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient(hunting): picked mon.noname-a con 0x7f573c09ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient(hunting): picked mon.noname-b con 0x7f573c0b12b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient(hunting): picked mon.noname-c con 0x7f573c1d7a40 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:57.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0b8000 con 0x7f573c09ac40 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c142160 con 0x7f573c0b12b0 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c142f70 con 0x7f573c1d7a40 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f574103e640 10 monclient(hunting): authenticate will time out at 924.106942s 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f573a59c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f573c0b12b0 0x7f573c1d7330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55758/0 (socket says 172.21.15.79:55758) 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f573a59c640 1 -- 172.21.15.79:0/2028103979 learned_addr learned my addr 172.21.15.79:0/2028103979 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.936+0000 7f573ad9d640 1 --1- 172.21.15.79:0/2028103979 >> v1:172.21.15.52:6789/0 conn(0x7f573c09ac40 0x7f573c142860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.79:53308/0 (socket says 172.21.15.79:53308) 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.?.objecter ms_handle_connect 0x7f573c0b12b0 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.?.objecter ms_handle_connect 0x7f573c09ac40 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.?.objecter ms_handle_connect 0x7f573c1d7a40 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185509229 0 0) 0x7f573c142160 con 0x7f573c0b12b0 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728004eb0 con 0x7f573c0b12b0 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821843118 0 0) 0x7f573c0b8000 con 0x7f573c09ac40 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c142160 con 0x7f573c09ac40 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794370824 0 0) 0x7f573c142f70 con 0x7f573c1d7a40 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c0b8000 con 0x7f573c1d7a40 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2489243577 0 0) 0x7f5728004eb0 con 0x7f573c0b12b0 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f573c142f70 con 0x7f573c0b12b0 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5730001980 con 0x7f573c0b12b0 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): dump: 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.928 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576402181 0 0) 0x7f573c142160 con 0x7f573c09ac40 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5728004eb0 con 0x7f573c09ac40 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3156002884 0 0) 0x7f573c0b8000 con 0x7f573c1d7a40 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f573c142160 con 0x7f573c1d7a40 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f572c001b80 con 0x7f573c09ac40 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): dump: 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.929 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5734001b40 con 0x7f573c1d7a40 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient(hunting): dump: 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.930 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 468650518 0 0) 0x7f573c142f70 con 0x7f573c0b12b0 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 >> v1:172.21.15.52:6790/0 conn(0x7f573c1d7a40 legacy=0x7f573c1e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 >> v1:172.21.15.52:6789/0 conn(0x7f573c09ac40 legacy=0x7f573c142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 monclient: found mon.b 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c1d7ed0 con 0x7f573c0b12b0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 10 monclient: _renew_subs 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c1e3400 con 0x7f573c0b12b0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 10 monclient: _renew_subs 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c1e3770 con 0x7f573c0b12b0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f574103e640 1 librados: init done 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5730001c40 con 0x7f573c0b12b0 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5730003a60 con 0x7f573c0b12b0 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f573959a640 4 set_mon_vals no callback set 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 monclient: dump: 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:57.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5730004010 con 0x7f573c0b12b0 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3086791822 0 0) 0x7f57300066e0 con 0x7f573c0b12b0 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 10 client.4108.objecter ms_dispatch 0x7f573c000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 3 client.4108.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 3 client.4108.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.937+0000 7f57237fe640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.938+0000 7f574103e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.938+0000 7f574103e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.938+0000 7f574103e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.938+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c005080 con 0x7f573c0b12b0 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.940+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f57300069c0 con 0x7f573c0b12b0 2024-03-29T11:19:57.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.940+0000 7f57237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:57.940+0000 7f57237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.013+0000 7f574103e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b7490542-1dbf-4ecd-96e0-1008fc1c34a2", "id": 1}] 2024-03-29T11:19:57.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.013+0000 7f574103e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b7490542-1dbf-4ecd-96e0-1008fc1c34a2", "id": 1}] 2024-03-29T11:19:57.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.013+0000 7f574103e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:57.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.013+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b7490542-1dbf-4ecd-96e0-1008fc1c34a2", "id": 1} v 0) v1 -- 0x7f571c0058b0 con 0x7f573c0b12b0 2024-03-29T11:19:57.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.018+0000 7f57237fe640 1 -- 172.21.15.79:0/2028103979 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7490542-1dbf-4ecd-96e0-1008fc1c34a2", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2353989388 0 940483276) 0x7f57300057c0 con 0x7f573c0b12b0 2024-03-29T11:19:57.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.018+0000 7f57237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b7490542-1dbf-4ecd-96e0-1008fc1c34a2", "id": 1}] 2024-03-29T11:19:57.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.018+0000 7f57237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.995 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 10 librados: watch_flush enter 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 10 librados: watch_flush exit 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 10 client.4108.objecter successfully canceled tick 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 10 monclient: shutdown 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 >> v1:172.21.15.79:6789/0 conn(0x7f573c0b12b0 legacy=0x7f573c1d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 shutdown_connections 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 >> 172.21.15.79:0/2028103979 conn(0x7f573c066b00 msgr2=0x7f573c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 shutdown_connections 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 1 -- 172.21.15.79:0/2028103979 wait complete. 2024-03-29T11:19:57.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.019+0000 7f574103e640 1 librados: shutdown 2024-03-29T11:19:58.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 7dfc37ef-c9dd-4263-9eab-c406f16723c0 2 2024-03-29T11:19:58.101 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.125+0000 7fd29afb6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.129+0000 7fc2ac6ba640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.129+0000 7fc2ac6ba640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.129+0000 7fc2ac6ba640 10 monclient: monmap: 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:58.130424+0000 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:58.130424+0000 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:58.106 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.114 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.132+0000 7f250a041040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 1 Processor -- start 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 1 -- start start 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 10 monclient: init 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a409ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.130+0000 7fc2ac6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a409c870 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a40b4360 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a409dec0 con 0x7fc2a40b4360 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a409da30 con 0x7fc2a409ac40 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a40b39c0 con 0x7fc2a409c870 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 10 monclient(hunting): authenticate will time out at 924.301126s 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2aa456640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc2a409ac40 0x7fc2a40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55766/0 (socket says 172.21.15.79:55766) 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2aa456640 1 -- 172.21.15.79:0/858859263 learned_addr learned my addr 172.21.15.79:0/858859263 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502082390 0 0) 0x7fc2a409da30 con 0x7fc2a409ac40 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc290003520 con 0x7fc2a409ac40 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557859877 0 0) 0x7fc29c001490 con 0x7fc2a40b4360 2024-03-29T11:19:58.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a409da30 con 0x7fc2a40b4360 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418722690 0 0) 0x7fc2a40b39c0 con 0x7fc2a409c870 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc29c001490 con 0x7fc2a409c870 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3391294380 0 0) 0x7fc290003520 con 0x7fc2a409ac40 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 >> v1:172.21.15.52:6790/0 conn(0x7fc2a409c870 legacy=0x7fc2a413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 >> v1:172.21.15.52:6789/0 conn(0x7fc2a40b4360 legacy=0x7fc2a414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 monclient: found mon.noname-c 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a40b5160 con 0x7fc2a409ac40 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc2a0001690 con 0x7fc2a409ac40 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: dump: 2024-03-29T11:19:58.119 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2a00017f0 con 0x7fc2a409ac40 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 1 -- 172.21.15.79:0/858859263 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc2a0001c00 con 0x7fc2a409ac40 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2ac6ba640 4 set_mon_vals no callback set 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.131+0000 7fc2a9454640 10 monclient: dump: 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 10 monclient: shutdown 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/858859263 >> v1:172.21.15.79:6789/0 conn(0x7fc2a409ac40 legacy=0x7fc2a40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/858859263 shutdown_connections 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/858859263 >> 172.21.15.79:0/858859263 conn(0x7fc2a4066b00 msgr2=0x7fc2a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/858859263 shutdown_connections 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.132+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/858859263 wait complete. 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient: monmap: 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 librados: starting msgr at 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 librados: starting objecter 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 Processor -- start 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 -- start start 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 librados: setting wanted keys 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 librados: calling monclient init 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient: init 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a409ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a40b0210 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a41d7c90 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a40afb50 con 0x7fc2a409ac40 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a41d70f0 con 0x7fc2a40b0210 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a40b0650 con 0x7fc2a41d7c90 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.133+0000 7fc2ac6ba640 10 monclient(hunting): authenticate will time out at 924.304061s 2024-03-29T11:19:58.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2a9c55640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc2a40b0210 0x7fc2a41d7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55768/0 (socket says 172.21.15.79:55768) 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2a9c55640 1 -- 172.21.15.79:0/1204371695 learned_addr learned my addr 172.21.15.79:0/1204371695 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.?.objecter ms_handle_connect 0x7fc2a40b0210 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.?.objecter ms_handle_connect 0x7fc2a409ac40 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.?.objecter ms_handle_connect 0x7fc2a41d7c90 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114137799 0 0) 0x7fc2a41d70f0 con 0x7fc2a40b0210 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc288004eb0 con 0x7fc2a40b0210 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486094295 0 0) 0x7fc2a40afb50 con 0x7fc2a409ac40 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a41d70f0 con 0x7fc2a409ac40 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643988283 0 0) 0x7fc2a40b0650 con 0x7fc2a41d7c90 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a40afb50 con 0x7fc2a41d7c90 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374132955 0 0) 0x7fc2a41d70f0 con 0x7fc2a409ac40 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2a40b0650 con 0x7fc2a409ac40 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc2a0001b00 con 0x7fc2a409ac40 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:58.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): dump: 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693864897 0 0) 0x7fc288004eb0 con 0x7fc2a40b0210 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2a41d70f0 con 0x7fc2a40b0210 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc294001900 con 0x7fc2a40b0210 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): dump: 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1558409897 0 0) 0x7fc2a40afb50 con 0x7fc2a41d7c90 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc288004eb0 con 0x7fc2a41d7c90 2024-03-29T11:19:58.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc29c001bc0 con 0x7fc2a41d7c90 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient(hunting): dump: 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2024827789 0 0) 0x7fc2a40b0650 con 0x7fc2a409ac40 2024-03-29T11:19:58.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 >> v1:172.21.15.52:6790/0 conn(0x7fc2a41d7c90 legacy=0x7fc2a41e1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 >> v1:172.21.15.79:6789/0 conn(0x7fc2a40b0210 legacy=0x7fc2a41d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 monclient: found mon.a 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a40acd00 con 0x7fc2a409ac40 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 5 monclient: authenticate success, global_id 4131 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 10 monclient: _renew_subs 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a40b18a0 con 0x7fc2a409ac40 2024-03-29T11:19:58.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 10 monclient: _renew_subs 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a41e35d0 con 0x7fc2a409ac40 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2ac6ba640 1 librados: init done 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2a0004a90 con 0x7fc2a409ac40 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc2a0004da0 con 0x7fc2a409ac40 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 monclient: dump: 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.130 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc2a8c53640 4 set_mon_vals no callback set 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc2a0005290 con 0x7fc2a409ac40 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1261967877 0 0) 0x7fc2a00067f0 con 0x7fc2a409ac40 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 10 client.4131.objecter ms_dispatch 0x7fc2a4000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.134+0000 7fc29affd640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.135+0000 7fc29affd640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.135+0000 7fc2ac6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.135+0000 7fc2ac6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.135+0000 7fc2ac6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:58.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.135+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c005080 con 0x7fc2a409ac40 2024-03-29T11:19:58.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.137+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc2a0006480 con 0x7fc2a409ac40 2024-03-29T11:19:58.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.137+0000 7fc29affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.137+0000 7fc29affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.158 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.182+0000 7fd29afb6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:58.165 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.188+0000 7f250a041040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:58.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.210+0000 7fc2ac6ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7dfc37ef-c9dd-4263-9eab-c406f16723c0", "id": 2}] 2024-03-29T11:19:58.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.210+0000 7fc2ac6ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7dfc37ef-c9dd-4263-9eab-c406f16723c0", "id": 2}] 2024-03-29T11:19:58.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.210+0000 7fc2ac6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:58.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.210+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7dfc37ef-c9dd-4263-9eab-c406f16723c0", "id": 2} v 0) v1 -- 0x7fc28c0058b0 con 0x7fc2a409ac40 2024-03-29T11:19:58.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.211+0000 7fc29affd640 1 -- 172.21.15.79:0/1204371695 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7dfc37ef-c9dd-4263-9eab-c406f16723c0", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1575440069 0 216599125) 0x7fc2a00069b0 con 0x7fc2a409ac40 2024-03-29T11:19:58.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc29affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7dfc37ef-c9dd-4263-9eab-c406f16723c0", "id": 2}] 2024-03-29T11:19:58.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc29affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.188 INFO:teuthology.orchestra.run.smithi079.stdout:2 2024-03-29T11:19:58.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 10 librados: watch_flush enter 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 10 librados: watch_flush exit 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 10 client.4131.objecter successfully canceled tick 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 10 monclient: shutdown 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 >> v1:172.21.15.52:6789/0 conn(0x7fc2a409ac40 legacy=0x7fc2a40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 shutdown_connections 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 >> 172.21.15.79:0/1204371695 conn(0x7fc2a4066b00 msgr2=0x7fc2a40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 shutdown_connections 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 1 -- 172.21.15.79:0/1204371695 wait complete. 2024-03-29T11:19:58.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.212+0000 7fc2ac6ba640 1 librados: shutdown 2024-03-29T11:19:58.199 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new ecaf2963-5d88-405a-b213-5bca421a9efa 3 2024-03-29T11:19:58.205 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.229+0000 7fd29afb6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:58.211 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.235+0000 7f250a041040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:58.262 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.286+0000 7fd29afb6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:58.267 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.290+0000 7f250a041040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.322+0000 7f043d6cf640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.322+0000 7f043d6cf640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.322+0000 7f043d6cf640 10 monclient: monmap: 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:58.323490+0000 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:58.323490+0000 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.299 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.300 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:58.300 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:58.300 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 1 Processor -- start 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 1 -- start start 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient: init 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f04380b8240 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f04380b1f60 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f043809c8c0 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043809bd80 con 0x7f043809c8c0 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043809b960 con 0x7f04380b1f60 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04380b2ec0 con 0x7f04380b8240 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.323+0000 7f043d6cf640 10 monclient(hunting): authenticate will time out at 924.494042s 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f04367fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f04380b1f60 0x7f043809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55774/0 (socket says 172.21.15.79:55774) 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f04367fc640 1 -- 172.21.15.79:0/2996094678 learned_addr learned my addr 172.21.15.79:0/2996094678 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650789329 0 0) 0x7f043809b960 con 0x7f04380b1f60 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0420003520 con 0x7f04380b1f60 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715631960 0 0) 0x7f04380b2ec0 con 0x7f04380b8240 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f043809b960 con 0x7f04380b8240 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597159182 0 0) 0x7f043809bd80 con 0x7f043809c8c0 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04380b2ec0 con 0x7f043809c8c0 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4233483946 0 0) 0x7f0420003520 con 0x7f04380b1f60 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 >> v1:172.21.15.52:6790/0 conn(0x7f04380b8240 legacy=0x7f043813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 >> v1:172.21.15.52:6789/0 conn(0x7f043809c8c0 legacy=0x7f043814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043813fb60 con 0x7f04380b1f60 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f042c001690 con 0x7f04380b1f60 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f043d6cf640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f043d6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: dump: 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.312 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f043d6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f042c0018e0 con 0x7f04380b1f60 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f043d6cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f043d6cf640 4 set_mon_vals no callback set 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 1 -- 172.21.15.79:0/2996094678 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f042c001c70 con 0x7f04380b1f60 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.324+0000 7f0435ffb640 10 monclient: dump: 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.313 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 10 monclient: shutdown 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 1 -- 172.21.15.79:0/2996094678 >> v1:172.21.15.79:6789/0 conn(0x7f04380b1f60 legacy=0x7f043809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 1 -- 172.21.15.79:0/2996094678 shutdown_connections 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 1 -- 172.21.15.79:0/2996094678 >> 172.21.15.79:0/2996094678 conn(0x7f0438066b00 msgr2=0x7f0438066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 1 -- 172.21.15.79:0/2996094678 shutdown_connections 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 1 -- 172.21.15.79:0/2996094678 wait complete. 2024-03-29T11:19:58.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.325+0000 7f043d6cf640 10 monclient: monmap: 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 1 librados: starting msgr at 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 1 librados: starting objecter 2024-03-29T11:19:58.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 1 Processor -- start 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 1 -- start start 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 1 librados: setting wanted keys 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 1 librados: calling monclient init 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 10 monclient: init 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.326+0000 7f043d6cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f043809c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f04380b8240 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f0438141ca0 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043813c4a0 con 0x7f0438141ca0 2024-03-29T11:19:58.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0438135390 con 0x7f043809c8c0 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0438134580 con 0x7f04380b8240 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient(hunting): authenticate will time out at 924.497144s 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f0436ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f043809c8c0 0x7f0438134c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55788/0 (socket says 172.21.15.79:55788) 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f0436ffd640 1 -- 172.21.15.79:0/494347224 learned_addr learned my addr 172.21.15.79:0/494347224 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 client.?.objecter ms_handle_connect 0x7f043809c8c0 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 client.?.objecter ms_handle_connect 0x7f04380b8240 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 client.?.objecter ms_handle_connect 0x7f0438141ca0 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277200345 0 0) 0x7f0438135390 con 0x7f043809c8c0 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0424004eb0 con 0x7f043809c8c0 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104368590 0 0) 0x7f0438134580 con 0x7f04380b8240 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438135390 con 0x7f04380b8240 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513516558 0 0) 0x7f043813c4a0 con 0x7f0438141ca0 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438134580 con 0x7f0438141ca0 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311196099 0 0) 0x7f0424004eb0 con 0x7f043809c8c0 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f043813c4a0 con 0x7f043809c8c0 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f04280019b0 con 0x7f043809c8c0 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.319 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1302079263 0 0) 0x7f0438135390 con 0x7f04380b8240 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0424004eb0 con 0x7f04380b8240 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879127079 0 0) 0x7f0438134580 con 0x7f0438141ca0 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0438135390 con 0x7f0438141ca0 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f042c001a50 con 0x7f04380b8240 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.320 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0430001b50 con 0x7f0438141ca0 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.321 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 620302959 0 0) 0x7f043813c4a0 con 0x7f043809c8c0 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 >> v1:172.21.15.52:6790/0 conn(0x7f04380b8240 legacy=0x7f04381deee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 >> v1:172.21.15.52:6789/0 conn(0x7f0438141ca0 legacy=0x7f04381e2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 monclient: found mon.b 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438134860 con 0x7f043809c8c0 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f041f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient: _renew_subs 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0438132200 con 0x7f043809c8c0 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient: _renew_subs 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04381330d0 con 0x7f043809c8c0 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.327+0000 7f043d6cf640 1 librados: init done 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0428001c50 con 0x7f043809c8c0 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0428003a60 con 0x7f043809c8c0 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 10 monclient: dump: 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f04357fa640 4 set_mon_vals no callback set 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0428004010 con 0x7f043809c8c0 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1208473612 0 0) 0x7f0428006780 con 0x7f043809c8c0 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 10 client.4120.objecter ms_dispatch 0x7f0438000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 3 client.4120.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:58.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 3 client.4120.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f041f7fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f043d6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f043d6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f043d6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.328+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0438132d10 con 0x7f043809c8c0 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.330+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f04280062f0 con 0x7f043809c8c0 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.330+0000 7f041f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.330+0000 7f041f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.325 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.331+0000 7fd29afb6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:58.325 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.334+0000 7f250a041040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:58.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.402+0000 7f043d6cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ecaf2963-5d88-405a-b213-5bca421a9efa", "id": 3}] 2024-03-29T11:19:58.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.402+0000 7f043d6cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ecaf2963-5d88-405a-b213-5bca421a9efa", "id": 3}] 2024-03-29T11:19:58.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.402+0000 7f043d6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.402+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ecaf2963-5d88-405a-b213-5bca421a9efa", "id": 3} v 0) v1 -- 0x7f043804f730 con 0x7f043809c8c0 2024-03-29T11:19:58.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.406+0000 7f041f7fe640 1 -- 172.21.15.79:0/494347224 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecaf2963-5d88-405a-b213-5bca421a9efa", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 256556529 0 525046306) 0x7f0428006590 con 0x7f043809c8c0 2024-03-29T11:19:58.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.406+0000 7f041f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ecaf2963-5d88-405a-b213-5bca421a9efa", "id": 3}] 2024-03-29T11:19:58.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.406+0000 7f041f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.383 INFO:teuthology.orchestra.run.smithi079.stdout:3 2024-03-29T11:19:58.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 10 librados: watch_flush enter 2024-03-29T11:19:58.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 10 librados: watch_flush exit 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 10 monclient: shutdown 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 >> v1:172.21.15.79:6789/0 conn(0x7f043809c8c0 legacy=0x7f0438134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 shutdown_connections 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 >> 172.21.15.79:0/494347224 conn(0x7f0438066b00 msgr2=0x7f043806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 shutdown_connections 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 1 -- 172.21.15.79:0/494347224 wait complete. 2024-03-29T11:19:58.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.407+0000 7f043d6cf640 1 librados: shutdown 2024-03-29T11:19:58.394 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48 4 2024-03-29T11:19:58.428 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.452+0000 7fd29afb6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:58.438 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.461+0000 7f250a041040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:58.502 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.525+0000 7fd29afb6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:58.509 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.533+0000 7f250a041040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:58.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.551+0000 7f014fbef640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.551+0000 7f014fbef640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.551+0000 7f014fbef640 10 monclient: monmap: 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:58.551908+0000 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:58.551908+0000 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.551+0000 7f014fbef640 1 Processor -- start 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.551+0000 7f014fbef640 1 -- start start 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.551+0000 7f014fbef640 10 monclient: init 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient(hunting): picked mon.noname-a con 0x7f014809ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient(hunting): picked mon.noname-b con 0x7f014809c870 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient(hunting): picked mon.noname-c con 0x7f01480b4360 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014809dec0 con 0x7f014809ac40 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014809da30 con 0x7f01480b4360 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480b39c0 con 0x7f014809c870 2024-03-29T11:19:58.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014fbef640 10 monclient(hunting): authenticate will time out at 924.722565s 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014e18c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f01480b4360 0x7f014814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55796/0 (socket says 172.21.15.79:55796) 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014e18c640 1 -- 172.21.15.79:0/1826945393 learned_addr learned my addr 172.21.15.79:0/1826945393 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205608445 0 0) 0x7f014809da30 con 0x7f01480b4360 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0134003520 con 0x7f01480b4360 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194548542 0 0) 0x7f01480b39c0 con 0x7f014809c870 2024-03-29T11:19:58.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014809da30 con 0x7f014809c870 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603606411 0 0) 0x7f0144001110 con 0x7f014809ac40 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.552+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01480b39c0 con 0x7f014809ac40 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107546385 0 0) 0x7f0134003520 con 0x7f01480b4360 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 >> v1:172.21.15.52:6790/0 conn(0x7f014809c870 legacy=0x7f014813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 >> v1:172.21.15.52:6789/0 conn(0x7f014809ac40 legacy=0x7f01480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 monclient: found mon.noname-c 2024-03-29T11:19:58.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480b5160 con 0x7f01480b4360 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0140001a10 con 0x7f01480b4360 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: dump: 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.540 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0140001bf0 con 0x7f01480b4360 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 1 -- 172.21.15.79:0/1826945393 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0140002000 con 0x7f01480b4360 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 4 set_mon_vals no callback set 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014c989640 10 monclient: dump: 2024-03-29T11:19:58.541 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 10 monclient: shutdown 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 1 -- 172.21.15.79:0/1826945393 >> v1:172.21.15.79:6789/0 conn(0x7f01480b4360 legacy=0x7f014814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 1 -- 172.21.15.79:0/1826945393 shutdown_connections 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 1 -- 172.21.15.79:0/1826945393 >> 172.21.15.79:0/1826945393 conn(0x7f0148066b00 msgr2=0x7f0148066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 1 -- 172.21.15.79:0/1826945393 shutdown_connections 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.553+0000 7f014fbef640 1 -- 172.21.15.79:0/1826945393 wait complete. 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 10 monclient: monmap: 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 1 librados: starting msgr at 2024-03-29T11:19:58.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 1 librados: starting objecter 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 1 Processor -- start 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 1 -- start start 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 1 librados: setting wanted keys 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 1 librados: calling monclient init 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.554+0000 7f014fbef640 10 monclient: init 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient(hunting): picked mon.noname-a con 0x7f014809ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient(hunting): picked mon.noname-c con 0x7f01480aee50 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient(hunting): picked mon.noname-b con 0x7f01480ae810 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01481d7b20 con 0x7f014809ac40 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480b00d0 con 0x7f01480ae810 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480ada70 con 0x7f01480aee50 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014fbef640 10 monclient(hunting): authenticate will time out at 924.725483s 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014e18c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f01480ae810 0x7f01480b10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55808/0 (socket says 172.21.15.79:55808) 2024-03-29T11:19:58.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f014e18c640 1 -- 172.21.15.79:0/2241287103 learned_addr learned my addr 172.21.15.79:0/2241287103 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 client.?.objecter ms_handle_connect 0x7f01480ae810 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 client.?.objecter ms_handle_connect 0x7f014809ac40 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 client.?.objecter ms_handle_connect 0x7f01480aee50 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815918582 0 0) 0x7f01480b00d0 con 0x7f01480ae810 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c004eb0 con 0x7f01480ae810 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163224524 0 0) 0x7f01480ada70 con 0x7f01480aee50 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01480b00d0 con 0x7f01480aee50 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079775860 0 0) 0x7f01481d7b20 con 0x7f014809ac40 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01480ada70 con 0x7f014809ac40 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3032312946 0 0) 0x7f012c004eb0 con 0x7f01480ae810 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01481d7b20 con 0x7f01480ae810 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0140001f40 con 0x7f01480ae810 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:58.546 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.555+0000 7f013a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3661056429 0 0) 0x7f01480b00d0 con 0x7f01480aee50 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f012c004eb0 con 0x7f01480aee50 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f013c001930 con 0x7f01480aee50 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3158664456 0 0) 0x7f01480ada70 con 0x7f014809ac40 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01480b00d0 con 0x7f014809ac40 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f01440018e0 con 0x7f014809ac40 2024-03-29T11:19:58.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2714048791 0 0) 0x7f01481d7b20 con 0x7f01480ae810 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 >> v1:172.21.15.52:6790/0 conn(0x7f01480aee50 legacy=0x7f01480b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 >> v1:172.21.15.52:6789/0 conn(0x7f014809ac40 legacy=0x7f01481d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 monclient: found mon.b 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01481d7640 con 0x7f01480ae810 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 10 monclient: _renew_subs 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01480b17b0 con 0x7f01480ae810 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 10 monclient: _renew_subs 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01480b1b50 con 0x7f01480ae810 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f014fbef640 1 librados: init done 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0140004e10 con 0x7f01480ae810 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f01400015b0 con 0x7f01480ae810 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013bfff640 4 set_mon_vals no callback set 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 monclient: dump: 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.551 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f01400054f0 con 0x7f01480ae810 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2955051211 0 0) 0x7f0140006c30 con 0x7f01480ae810 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 10 client.4126.objecter ms_dispatch 0x7f0148000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 3 client.4126.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 3 client.4126.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.556+0000 7f013a7fc640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.557+0000 7f014fbef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.557+0000 7f014fbef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.557+0000 7f014fbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.557+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130005080 con 0x7f01480ae810 2024-03-29T11:19:58.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.559+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f01400067d0 con 0x7f01480ae810 2024-03-29T11:19:58.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.559+0000 7f013a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.559+0000 7f013a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.553 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.574+0000 7fd29afb6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:58.557 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.581+0000 7f250a041040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:58.605 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.628+0000 7fd29afb6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:58.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.631+0000 7f014fbef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48", "id": 4}] 2024-03-29T11:19:58.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.631+0000 7f014fbef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48", "id": 4}] 2024-03-29T11:19:58.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.631+0000 7f014fbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.631+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48", "id": 4} v 0) v1 -- 0x7f01300058b0 con 0x7f01480ae810 2024-03-29T11:19:58.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f013a7fc640 1 -- 172.21.15.79:0/2241287103 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3384231617 0 1697012583) 0x7f0140006a40 con 0x7f01480ae810 2024-03-29T11:19:58.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f013a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48", "id": 4}] 2024-03-29T11:19:58.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f013a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.612 INFO:teuthology.orchestra.run.smithi079.stdout:4 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 10 librados: watch_flush enter 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 10 librados: watch_flush exit 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 10 monclient: shutdown 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 >> v1:172.21.15.79:6789/0 conn(0x7f01480ae810 legacy=0x7f01480b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 shutdown_connections 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.635+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 >> 172.21.15.79:0/2241287103 conn(0x7f0148066b00 msgr2=0x7f014813a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.636+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 shutdown_connections 2024-03-29T11:19:58.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.636+0000 7f014fbef640 1 -- 172.21.15.79:0/2241287103 wait complete. 2024-03-29T11:19:58.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.636+0000 7f014fbef640 1 librados: shutdown 2024-03-29T11:19:58.614 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.636+0000 7f250a041040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:58.622 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new c780a8b9-2eba-4d45-9693-5af6ad16610e 5 2024-03-29T11:19:58.656 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:58.679+0000 7fd29afb6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:58.661 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:58.684+0000 7f250a041040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:58.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.744+0000 7fa83a13b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.744+0000 7fa83a13b640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.744+0000 7fa83a13b640 10 monclient: monmap: 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:58.745756+0000 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:58.745756+0000 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:58.722 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.745+0000 7fa83a13b640 1 Processor -- start 2024-03-29T11:19:58.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.745+0000 7fa83a13b640 1 -- start start 2024-03-29T11:19:58.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.745+0000 7fa83a13b640 10 monclient: init 2024-03-29T11:19:58.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient(hunting): picked mon.noname-a con 0x7fa83409ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient(hunting): picked mon.noname-b con 0x7fa83409c870 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8340b4360 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83409dec0 con 0x7fa83409ac40 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83409da30 con 0x7fa8340b4360 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8340b39c0 con 0x7fa83409c870 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa83a13b640 10 monclient(hunting): authenticate will time out at 924.916472s 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa833fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fa8340b4360 0x7fa83414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55816/0 (socket says 172.21.15.79:55816) 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa833fff640 1 -- 172.21.15.79:0/3153981637 learned_addr learned my addr 172.21.15.79:0/3153981637 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304551293 0 0) 0x7fa83409da30 con 0x7fa8340b4360 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820003520 con 0x7fa8340b4360 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880042804 0 0) 0x7fa824001110 con 0x7fa83409ac40 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa83409da30 con 0x7fa83409ac40 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588705119 0 0) 0x7fa8340b39c0 con 0x7fa83409c870 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa824001110 con 0x7fa83409c870 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4208819179 0 0) 0x7fa820003520 con 0x7fa8340b4360 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 >> v1:172.21.15.52:6790/0 conn(0x7fa83409c870 legacy=0x7fa83413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.746+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 >> v1:172.21.15.52:6789/0 conn(0x7fa83409ac40 legacy=0x7fa8340b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340b5160 con 0x7fa8340b4360 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:58.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa82c001a10 con 0x7fa8340b4360 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: dump: 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.735 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa82c001bf0 con 0x7fa8340b4360 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 4 set_mon_vals no callback set 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 1 -- 172.21.15.79:0/3153981637 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa82c002000 con 0x7fa8340b4360 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa832ffd640 10 monclient: dump: 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.736 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 10 monclient: shutdown 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 1 -- 172.21.15.79:0/3153981637 >> v1:172.21.15.79:6789/0 conn(0x7fa8340b4360 legacy=0x7fa83414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 1 -- 172.21.15.79:0/3153981637 shutdown_connections 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 1 -- 172.21.15.79:0/3153981637 >> 172.21.15.79:0/3153981637 conn(0x7fa834066b00 msgr2=0x7fa834066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 1 -- 172.21.15.79:0/3153981637 shutdown_connections 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.747+0000 7fa83a13b640 1 -- 172.21.15.79:0/3153981637 wait complete. 2024-03-29T11:19:58.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 10 monclient: monmap: 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 1 librados: starting msgr at 2024-03-29T11:19:58.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 1 librados: starting objecter 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 1 Processor -- start 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 1 -- start start 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 1 librados: setting wanted keys 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 1 librados: calling monclient init 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.748+0000 7fa83a13b640 10 monclient: init 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient(hunting): picked mon.noname-a con 0x7fa83409ac40 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient(hunting): picked mon.noname-c con 0x7fa834142c20 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8341d7dc0 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8341d4d30 con 0x7fa83409ac40 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8341430b0 con 0x7fa8341d7dc0 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8341d71b0 con 0x7fa834142c20 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa83a13b640 10 monclient(hunting): authenticate will time out at 924.919476s 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa833fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fa8341d7dc0 0x7fa834145a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55828/0 (socket says 172.21.15.79:55828) 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa833fff640 1 -- 172.21.15.79:0/1574891665 learned_addr learned my addr 172.21.15.79:0/1574891665 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 client.?.objecter ms_handle_connect 0x7fa8341d7dc0 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 client.?.objecter ms_handle_connect 0x7fa834142c20 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 client.?.objecter ms_handle_connect 0x7fa83409ac40 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633760010 0 0) 0x7fa8341430b0 con 0x7fa8341d7dc0 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:58.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa814004eb0 con 0x7fa8341d7dc0 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495442189 0 0) 0x7fa8341d71b0 con 0x7fa834142c20 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8341430b0 con 0x7fa834142c20 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018001105 0 0) 0x7fa8341d4d30 con 0x7fa83409ac40 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8341d71b0 con 0x7fa83409ac40 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659225153 0 0) 0x7fa814004eb0 con 0x7fa8341d7dc0 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8341d4d30 con 0x7fa8341d7dc0 2024-03-29T11:19:58.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa82c001f40 con 0x7fa8341d7dc0 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.749+0000 7fa830ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4162507447 0 0) 0x7fa8341430b0 con 0x7fa834142c20 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa814004eb0 con 0x7fa834142c20 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa81c001870 con 0x7fa834142c20 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.743 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936543993 0 0) 0x7fa8341d71b0 con 0x7fa83409ac40 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8341430b0 con 0x7fa83409ac40 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa8240018a0 con 0x7fa83409ac40 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.744 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377890919 0 0) 0x7fa8341d4d30 con 0x7fa8341d7dc0 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 >> v1:172.21.15.52:6790/0 conn(0x7fa834142c20 legacy=0x7fa8341d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 >> v1:172.21.15.52:6789/0 conn(0x7fa83409ac40 legacy=0x7fa834143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 monclient: found mon.b 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834144ce0 con 0x7fa8341d7dc0 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:58.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 10 monclient: _renew_subs 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa834145490 con 0x7fa8341d7dc0 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 10 monclient: _renew_subs 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa834146330 con 0x7fa8341d7dc0 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa83a13b640 1 librados: init done 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa82c004e10 con 0x7fa8341d7dc0 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa82c005120 con 0x7fa8341d7dc0 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 monclient: dump: 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa8327fc640 4 set_mon_vals no callback set 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa82c005610 con 0x7fa8341d7dc0 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3315695837 0 0) 0x7fa82c001570 con 0x7fa8341d7dc0 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 10 client.4132.objecter ms_dispatch 0x7fa834000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 3 client.4132.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:58.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.750+0000 7fa830ff9640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.751+0000 7fa83a13b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.751+0000 7fa83a13b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.751+0000 7fa83a13b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.751+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa818005080 con 0x7fa8341d7dc0 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.753+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa82c006020 con 0x7fa8341d7dc0 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.753+0000 7fa830ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.753+0000 7fa830ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.823+0000 7fa83a13b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c780a8b9-2eba-4d45-9693-5af6ad16610e", "id": 5}] 2024-03-29T11:19:58.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.823+0000 7fa83a13b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c780a8b9-2eba-4d45-9693-5af6ad16610e", "id": 5}] 2024-03-29T11:19:58.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.823+0000 7fa83a13b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.823+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c780a8b9-2eba-4d45-9693-5af6ad16610e", "id": 5} v 0) v1 -- 0x7fa8180058b0 con 0x7fa8341d7dc0 2024-03-29T11:19:58.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.827+0000 7fa830ff9640 1 -- 172.21.15.79:0/1574891665 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c780a8b9-2eba-4d45-9693-5af6ad16610e", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 761471140 0 1988412176) 0x7fa82c005990 con 0x7fa8341d7dc0 2024-03-29T11:19:58.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.827+0000 7fa830ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c780a8b9-2eba-4d45-9693-5af6ad16610e", "id": 5}] 2024-03-29T11:19:58.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.827+0000 7fa830ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.804 INFO:teuthology.orchestra.run.smithi079.stdout:5 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 10 librados: watch_flush enter 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 10 librados: watch_flush exit 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 10 monclient: shutdown 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 >> v1:172.21.15.79:6789/0 conn(0x7fa8341d7dc0 legacy=0x7fa834145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 shutdown_connections 2024-03-29T11:19:58.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 >> 172.21.15.79:0/1574891665 conn(0x7fa834066b00 msgr2=0x7fa83413a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 shutdown_connections 2024-03-29T11:19:58.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 1 -- 172.21.15.79:0/1574891665 wait complete. 2024-03-29T11:19:58.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.828+0000 7fa83a13b640 1 librados: shutdown 2024-03-29T11:19:58.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 37614f81-d3d4-4fb9-b15f-1a4fe106d618 6 2024-03-29T11:19:58.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.979+0000 7fdd721e8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.979+0000 7fdd721e8640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.979+0000 7fdd721e8640 10 monclient: monmap: 2024-03-29T11:19:58.956 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:58.980654+0000 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:58.980654+0000 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:58.957 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.980+0000 7fdd721e8640 1 Processor -- start 2024-03-29T11:19:58.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.980+0000 7fdd721e8640 1 -- start start 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.980+0000 7fdd721e8640 10 monclient: init 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6c09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6c09c870 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6c0b4360 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c09dec0 con 0x7fdd6c09c870 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c09da30 con 0x7fdd6c09ac40 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c0b39c0 con 0x7fdd6c0b4360 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient(hunting): authenticate will time out at 925.151337s 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6b7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fdd6c09ac40 0x7fdd6c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55840/0 (socket says 172.21.15.79:55840) 2024-03-29T11:19:58.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6b7fe640 1 -- 172.21.15.79:0/2491207165 learned_addr learned my addr 172.21.15.79:0/2491207165 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259717621 0 0) 0x7fdd6c09da30 con 0x7fdd6c09ac40 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54003520 con 0x7fdd6c09ac40 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957224366 0 0) 0x7fdd6c0b39c0 con 0x7fdd6c0b4360 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c09da30 con 0x7fdd6c0b4360 2024-03-29T11:19:58.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359522917 0 0) 0x7fdd5c001110 con 0x7fdd6c09c870 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c0b39c0 con 0x7fdd6c09c870 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436931285 0 0) 0x7fdd54003520 con 0x7fdd6c09ac40 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 >> v1:172.21.15.52:6790/0 conn(0x7fdd6c0b4360 legacy=0x7fdd6c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 >> v1:172.21.15.52:6789/0 conn(0x7fdd6c09c870 legacy=0x7fdd6c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c0b5160 con 0x7fdd6c09ac40 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdd60001690 con 0x7fdd6c09ac40 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: dump: 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.970 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd60001870 con 0x7fdd6c09ac40 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd721e8640 4 set_mon_vals no callback set 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 1 -- 172.21.15.79:0/2491207165 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdd60001c80 con 0x7fdd6c09ac40 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.981+0000 7fdd6a7fc640 10 monclient: dump: 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 10 monclient: shutdown 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 1 -- 172.21.15.79:0/2491207165 >> v1:172.21.15.79:6789/0 conn(0x7fdd6c09ac40 legacy=0x7fdd6c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 1 -- 172.21.15.79:0/2491207165 shutdown_connections 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 1 -- 172.21.15.79:0/2491207165 >> 172.21.15.79:0/2491207165 conn(0x7fdd6c066b00 msgr2=0x7fdd6c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 1 -- 172.21.15.79:0/2491207165 shutdown_connections 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.982+0000 7fdd721e8640 1 -- 172.21.15.79:0/2491207165 wait complete. 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.983+0000 7fdd721e8640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.983+0000 7fdd721e8640 10 monclient: monmap: 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.973 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.983+0000 7fdd721e8640 1 librados: starting msgr at 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.983+0000 7fdd721e8640 1 librados: starting objecter 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.983+0000 7fdd721e8640 1 Processor -- start 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.984+0000 7fdd721e8640 1 -- start start 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.984+0000 7fdd721e8640 1 librados: setting wanted keys 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.984+0000 7fdd721e8640 1 librados: calling monclient init 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.984+0000 7fdd721e8640 10 monclient: init 2024-03-29T11:19:58.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.984+0000 7fdd721e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6c09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6c0b0fb0 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6c142cd0 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c141fd0 con 0x7fdd6c0b0fb0 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c142840 con 0x7fdd6c09ac40 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c13a7f0 con 0x7fdd6c142cd0 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient(hunting): authenticate will time out at 925.155157s 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd6b7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fdd6c09ac40 0x7fdd6c1da540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55842/0 (socket says 172.21.15.79:55842) 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd6b7fe640 1 -- 172.21.15.79:0/639057502 learned_addr learned my addr 172.21.15.79:0/639057502 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:58.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 client.?.objecter ms_handle_connect 0x7fdd6c09ac40 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 client.?.objecter ms_handle_connect 0x7fdd6c0b0fb0 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 client.?.objecter ms_handle_connect 0x7fdd6c142cd0 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596019384 0 0) 0x7fdd6c142840 con 0x7fdd6c09ac40 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd58004eb0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30588321 0 0) 0x7fdd6c141fd0 con 0x7fdd6c0b0fb0 2024-03-29T11:19:58.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c142840 con 0x7fdd6c0b0fb0 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765356073 0 0) 0x7fdd6c13a7f0 con 0x7fdd6c142cd0 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c141fd0 con 0x7fdd6c142cd0 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2854595698 0 0) 0x7fdd58004eb0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd6c13a7f0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdd60001bc0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.978 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259237248 0 0) 0x7fdd6c142840 con 0x7fdd6c0b0fb0 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd58004eb0 con 0x7fdd6c0b0fb0 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdd5c001860 con 0x7fdd6c0b0fb0 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:58.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1544207772 0 0) 0x7fdd6c141fd0 con 0x7fdd6c142cd0 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd6c142840 con 0x7fdd6c142cd0 2024-03-29T11:19:58.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdd640019f0 con 0x7fdd6c142cd0 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1007150904 0 0) 0x7fdd6c13a7f0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 >> v1:172.21.15.52:6790/0 conn(0x7fdd6c142cd0 legacy=0x7fdd6c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 >> v1:172.21.15.52:6789/0 conn(0x7fdd6c0b0fb0 legacy=0x7fdd6c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 monclient: found mon.b 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c1dac50 con 0x7fdd6c09ac40 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:58.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient: _renew_subs 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c0b1a50 con 0x7fdd6c09ac40 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient: _renew_subs 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c142520 con 0x7fdd6c09ac40 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd721e8640 1 librados: init done 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd60004bf0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdd60004f00 con 0x7fdd6c09ac40 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd53fff640 10 monclient: dump: 2024-03-29T11:19:58.983 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.985+0000 7fdd69ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdd600053f0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3885575549 0 0) 0x7fdd60006ad0 con 0x7fdd6c09ac40 2024-03-29T11:19:58.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd53fff640 10 client.4138.objecter ms_dispatch 0x7fdd6c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd53fff640 3 client.4138.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd53fff640 3 client.4138.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd53fff640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd721e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd721e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd721e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.986+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c005080 con 0x7fdd6c09ac40 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.988+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdd60006590 con 0x7fdd6c09ac40 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.988+0000 7fdd53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:58.988+0000 7fdd53fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:59.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.061+0000 7fdd721e8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "37614f81-d3d4-4fb9-b15f-1a4fe106d618", "id": 6}] 2024-03-29T11:19:59.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.061+0000 7fdd721e8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "37614f81-d3d4-4fb9-b15f-1a4fe106d618", "id": 6}] 2024-03-29T11:19:59.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.061+0000 7fdd721e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.061+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "37614f81-d3d4-4fb9-b15f-1a4fe106d618", "id": 6} v 0) v1 -- 0x7fdd4c0058b0 con 0x7fdd6c09ac40 2024-03-29T11:19:59.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd53fff640 1 -- 172.21.15.79:0/639057502 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37614f81-d3d4-4fb9-b15f-1a4fe106d618", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 857995944 0 1113811849) 0x7fdd60006800 con 0x7fdd6c09ac40 2024-03-29T11:19:59.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "37614f81-d3d4-4fb9-b15f-1a4fe106d618", "id": 6}] 2024-03-29T11:19:59.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd53fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:59.041 INFO:teuthology.orchestra.run.smithi079.stdout:6 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 10 librados: watch_flush enter 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 10 librados: watch_flush exit 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 10 monclient: shutdown 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 >> v1:172.21.15.79:6789/0 conn(0x7fdd6c09ac40 legacy=0x7fdd6c1da540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 shutdown_connections 2024-03-29T11:19:59.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.065+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 >> 172.21.15.79:0/639057502 conn(0x7fdd6c066b00 msgr2=0x7fdd6c0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.066+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 shutdown_connections 2024-03-29T11:19:59.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.066+0000 7fdd721e8640 1 -- 172.21.15.79:0/639057502 wait complete. 2024-03-29T11:19:59.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.066+0000 7fdd721e8640 1 librados: shutdown 2024-03-29T11:19:59.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 80826614-85ac-4fca-b118-837ebbdb3594 7 2024-03-29T11:19:59.055 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.079+0000 7fd29afb6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:59.057 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.081+0000 7f250a041040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.175+0000 7f9089286640 10 monclient: get_monmap_and_config 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.175+0000 7f9089286640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.175+0000 7f9089286640 10 monclient: monmap: 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:59.176590+0000 2024-03-29T11:19:59.152 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:59.176590+0000 2024-03-29T11:19:59.153 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.153 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.153 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:59.153 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:59.153 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:59.153 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 1 Processor -- start 2024-03-29T11:19:59.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 1 -- start start 2024-03-29T11:19:59.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient: init 2024-03-29T11:19:59.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient(hunting): picked mon.noname-c con 0x7f908409ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient(hunting): picked mon.noname-a con 0x7f908409c870 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient(hunting): picked mon.noname-b con 0x7f90840b4360 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908409dec0 con 0x7f908409c870 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908409da30 con 0x7f908409ac40 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840b39c0 con 0x7f90840b4360 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.176+0000 7f9089286640 10 monclient(hunting): authenticate will time out at 925.347060s 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9082ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f908409ac40 0x7f90840b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55852/0 (socket says 172.21.15.79:55852) 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9082ffd640 1 -- 172.21.15.79:0/1123667342 learned_addr learned my addr 172.21.15.79:0/1123667342 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:59.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953679527 0 0) 0x7f908409da30 con 0x7f908409ac40 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f906c003520 con 0x7f908409ac40 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265882127 0 0) 0x7f906c003520 con 0x7f908409ac40 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 >> v1:172.21.15.52:6790/0 conn(0x7f90840b4360 legacy=0x7f908414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 >> v1:172.21.15.52:6789/0 conn(0x7f908409c870 legacy=0x7f908413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840b5160 con 0x7f908409ac40 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9078001690 con 0x7f908409ac40 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9089286640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9089286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: dump: 2024-03-29T11:19:59.166 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9089286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90780017f0 con 0x7f908409ac40 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9089286640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 1 -- 172.21.15.79:0/1123667342 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9078001c40 con 0x7f908409ac40 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9089286640 4 set_mon_vals no callback set 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.177+0000 7f9081ffb640 10 monclient: dump: 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.168 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 10 monclient: shutdown 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 1 -- 172.21.15.79:0/1123667342 >> v1:172.21.15.79:6789/0 conn(0x7f908409ac40 legacy=0x7f90840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 1 -- 172.21.15.79:0/1123667342 shutdown_connections 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 1 -- 172.21.15.79:0/1123667342 >> 172.21.15.79:0/1123667342 conn(0x7f9084066b00 msgr2=0x7f9084066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 1 -- 172.21.15.79:0/1123667342 shutdown_connections 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 1 -- 172.21.15.79:0/1123667342 wait complete. 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.178+0000 7f9089286640 10 monclient: monmap: 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 librados: starting msgr at 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 librados: starting objecter 2024-03-29T11:19:59.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 Processor -- start 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 -- start start 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 librados: setting wanted keys 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 librados: calling monclient init 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient: init 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient(hunting): picked mon.noname-b con 0x7f908409ac40 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient(hunting): picked mon.noname-a con 0x7f9084143ac0 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient(hunting): picked mon.noname-c con 0x7f90841373e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084145cd0 con 0x7f9084143ac0 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084140de0 con 0x7f908409ac40 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084146420 con 0x7f90841373e0 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.179+0000 7f9089286640 10 monclient(hunting): authenticate will time out at 925.350021s 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9082ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f908409ac40 0x7f90841446b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55866/0 (socket says 172.21.15.79:55866) 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9082ffd640 1 -- 172.21.15.79:0/1557206170 learned_addr learned my addr 172.21.15.79:0/1557206170 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.?.objecter ms_handle_connect 0x7f908409ac40 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.?.objecter ms_handle_connect 0x7f90841373e0 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924245099 0 0) 0x7f9084140de0 con 0x7f908409ac40 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9070004eb0 con 0x7f908409ac40 2024-03-29T11:19:59.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.?.objecter ms_handle_connect 0x7f9084143ac0 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048476690 0 0) 0x7f9084146420 con 0x7f90841373e0 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9084140de0 con 0x7f90841373e0 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047297601 0 0) 0x7f9084145cd0 con 0x7f9084143ac0 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9084146420 con 0x7f9084143ac0 2024-03-29T11:19:59.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561802583 0 0) 0x7f9070004eb0 con 0x7f908409ac40 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9084145cd0 con 0x7f908409ac40 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9078001650 con 0x7f908409ac40 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.174 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3181682767 0 0) 0x7f9084140de0 con 0x7f90841373e0 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9070004eb0 con 0x7f90841373e0 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f907c001ac0 con 0x7f90841373e0 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.175 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 129098339 0 0) 0x7f9084145cd0 con 0x7f908409ac40 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 >> v1:172.21.15.52:6790/0 conn(0x7f90841373e0 legacy=0x7f9084141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 >> v1:172.21.15.52:6789/0 conn(0x7f9084143ac0 legacy=0x7f9084136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 monclient: found mon.b 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084141980 con 0x7f908409ac40 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 10 monclient: _renew_subs 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9084141c30 con 0x7f908409ac40 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 10 monclient: _renew_subs 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90841e3be0 con 0x7f908409ac40 2024-03-29T11:19:59.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f9089286640 1 librados: init done 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9078001cc0 con 0x7f908409ac40 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9078004e10 con 0x7f908409ac40 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f90817fa640 4 set_mon_vals no callback set 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 monclient: dump: 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.178 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f90780053c0 con 0x7f908409ac40 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 506954963 0 0) 0x7f9078006a40 con 0x7f908409ac40 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 10 client.4144.objecter ms_dispatch 0x7f9084000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 3 client.4144.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 3 client.4144.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.180+0000 7f906b7fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:19:59.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.181+0000 7f9089286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.181+0000 7f9089286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.182+0000 7f9089286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.182+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064005080 con 0x7f908409ac40 2024-03-29T11:19:59.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.183+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f90780063e0 con 0x7f908409ac40 2024-03-29T11:19:59.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.184+0000 7f906b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.184+0000 7f906b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:59.197 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.221+0000 7fd29afb6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:59.198 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.221+0000 7f250a041040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:59.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.257+0000 7f9089286640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "80826614-85ac-4fca-b118-837ebbdb3594", "id": 7}] 2024-03-29T11:19:59.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.257+0000 7f9089286640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "80826614-85ac-4fca-b118-837ebbdb3594", "id": 7}] 2024-03-29T11:19:59.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.257+0000 7f9089286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:19:59.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.257+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "80826614-85ac-4fca-b118-837ebbdb3594", "id": 7} v 0) v1 -- 0x7f90640058b0 con 0x7f908409ac40 2024-03-29T11:19:59.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.260+0000 7f906b7fe640 1 -- 172.21.15.79:0/1557206170 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "80826614-85ac-4fca-b118-837ebbdb3594", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4092546414 0 1371665406) 0x7f90780066d0 con 0x7f908409ac40 2024-03-29T11:19:59.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.260+0000 7f906b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "80826614-85ac-4fca-b118-837ebbdb3594", "id": 7}] 2024-03-29T11:19:59.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.260+0000 7f906b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:59.237 INFO:teuthology.orchestra.run.smithi079.stdout:7 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 10 librados: watch_flush enter 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 10 librados: watch_flush exit 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 10 monclient: shutdown 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 >> v1:172.21.15.79:6789/0 conn(0x7f908409ac40 legacy=0x7f90841446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 shutdown_connections 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 >> 172.21.15.79:0/1557206170 conn(0x7f9084066b00 msgr2=0x7f908413b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 shutdown_connections 2024-03-29T11:19:59.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 1 -- 172.21.15.79:0/1557206170 wait complete. 2024-03-29T11:19:59.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T11:19:59.262+0000 7f9089286640 1 librados: shutdown 2024-03-29T11:19:59.249 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:59.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:59.251 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:59.251 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:59.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:59.253 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:59.253 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:59.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:59.255 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:59.255 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:59.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:59.257 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:59.257 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:59.257 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:59.292 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:59.292 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:59.292 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:59.294 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:59.294 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:59.294 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:59.296 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:59.296 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:59.296 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:59.298 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:59.299 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:59.330 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.353+0000 7fd29afb6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:59.350 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.374+0000 7f250a041040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:59.394 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.418+0000 7fd29afb6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:59.400 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.423+0000 7f250a041040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:59.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.438+0000 7fe0a5d9f640 10 monclient: get_monmap_and_config 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.438+0000 7fe0a5d9f640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.438+0000 7fe0a5d9f640 10 monclient: monmap: 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:59.439455+0000 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:59.439455+0000 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:19:59.415 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 1 Processor -- start 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 1 -- start start 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient: init 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a0145610 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a0146050 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a013c030 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00aacc0 con 0x7fe0a0145610 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00b2720 con 0x7fe0a0146050 2024-03-29T11:19:59.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00aaf30 con 0x7fe0a013c030 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.439+0000 7fe0a5d9f640 10 monclient(hunting): authenticate will time out at 929.580060s 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0a4d9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe0a0145610 0x7fe0a00b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33436/0 (socket says 172.21.15.52:33436) 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0a4d9d640 1 -- 172.21.15.52:0/501026409 learned_addr learned my addr 172.21.15.52:0/501026409 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088035418 0 0) 0x7fe0a00aaf30 con 0x7fe0a013c030 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088003520 con 0x7fe0a013c030 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764064631 0 0) 0x7fe0a00aacc0 con 0x7fe0a0145610 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00aaf30 con 0x7fe0a0145610 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231145238 0 0) 0x7fe0a00b2720 con 0x7fe0a0146050 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient(hunting): my global_id is 4171 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00aacc0 con 0x7fe0a0146050 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 13743514 0 0) 0x7fe088003520 con 0x7fe0a013c030 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 >> v1:172.21.15.79:6789/0 conn(0x7fe0a0146050 legacy=0x7fe0a013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 >> v1:172.21.15.52:6789/0 conn(0x7fe0a0145610 legacy=0x7fe0a00b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 monclient: found mon.noname-b 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a00a9b20 con 0x7fe0a013c030 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe098001a10 con 0x7fe0a013c030 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: dump: 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.426 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0a5d9f640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0a5d9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe098001b70 con 0x7fe0a013c030 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0a5d9f640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 1 -- 172.21.15.52:0/501026409 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe098001f40 con 0x7fe0a013c030 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0a5d9f640 4 set_mon_vals no callback set 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.440+0000 7fe0977fe640 10 monclient: dump: 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.427 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 10 monclient: shutdown 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/501026409 >> v1:172.21.15.52:6790/0 conn(0x7fe0a013c030 legacy=0x7fe0a014d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/501026409 shutdown_connections 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/501026409 >> 172.21.15.52:0/501026409 conn(0x7fe0a00726e0 msgr2=0x7fe0a0066920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/501026409 shutdown_connections 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/501026409 wait complete. 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.441+0000 7fe0a5d9f640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 10 monclient: monmap: 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-29T11:19:59.428 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 1 librados: starting msgr at 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 1 librados: starting objecter 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 1 Processor -- start 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 1 -- start start 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 1 librados: setting wanted keys 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 1 librados: calling monclient init 2024-03-29T11:19:59.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.442+0000 7fe0a5d9f640 10 monclient: init 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a013c030 addr v1:172.21.15.79:6789/0 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a0145610 addr v1:172.21.15.52:6789/0 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a0146050 addr v1:172.21.15.52:6790/0 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00ac070 con 0x7fe0a0145610 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a013ec70 con 0x7fe0a013c030 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a013f190 con 0x7fe0a0146050 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a5d9f640 10 monclient(hunting): authenticate will time out at 929.583182s 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a559e640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe0a0146050 0x7fe0a01e28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58950/0 (socket says 172.21.15.52:58950) 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe097fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe0a0145610 0x7fe0a0136d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33438/0 (socket says 172.21.15.52:33438) 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0a559e640 1 -- 172.21.15.52:0/907610094 learned_addr learned my addr 172.21.15.52:0/907610094 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:19:59.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 client.?.objecter ms_handle_connect 0x7fe0a0146050 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 client.?.objecter ms_handle_connect 0x7fe0a0145610 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 client.?.objecter ms_handle_connect 0x7fe0a013c030 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072234435 0 0) 0x7fe0a013f190 con 0x7fe0a0146050 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c004eb0 con 0x7fe0a0146050 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761478964 0 0) 0x7fe0a00ac070 con 0x7fe0a0145610 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a013f190 con 0x7fe0a0145610 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505623562 0 0) 0x7fe0a013ec70 con 0x7fe0a013c030 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00ac070 con 0x7fe0a013c030 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774852150 0 0) 0x7fe0a013f190 con 0x7fe0a0145610 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a013ec70 con 0x7fe0a0145610 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378265727 0 0) 0x7fe08c004eb0 con 0x7fe0a0146050 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a013f190 con 0x7fe0a0146050 2024-03-29T11:19:59.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2899593659 0 0) 0x7fe0a00ac070 con 0x7fe0a013c030 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe08c004eb0 con 0x7fe0a013c030 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe090001850 con 0x7fe0a0145610 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): dump: 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.433 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe0980018d0 con 0x7fe0a0146050 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): dump: 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.434 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe09c0018b0 con 0x7fe0a013c030 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 10 monclient(hunting): dump: 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.435 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512344193 0 0) 0x7fe0a013ec70 con 0x7fe0a0145610 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 >> v1:172.21.15.52:6790/0 conn(0x7fe0a0146050 legacy=0x7fe0a01e28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.443+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 >> v1:172.21.15.79:6789/0 conn(0x7fe0a013c030 legacy=0x7fe0a01d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 1 monclient: found mon.a 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a013f320 con 0x7fe0a0145610 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 10 monclient: _renew_subs 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:59.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a01d76c0 con 0x7fe0a0145610 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 10 monclient: _renew_subs 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a013e930 con 0x7fe0a0145610 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0a5d9f640 1 librados: init done 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0900019b0 con 0x7fe0a0145610 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe090003790 con 0x7fe0a0145610 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe096ffd640 4 set_mon_vals no callback set 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 monclient: dump: 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-29T11:19:59.437 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe090003d40 con 0x7fe0a0145610 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fe090006800 con 0x7fe0a0145610 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 10 client.4191.objecter ms_dispatch 0x7fe0a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:59.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.444+0000 7fe0957fa640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.445+0000 7fe0a5d9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.445+0000 7fe0a5d9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.445+0000 7fe0a5d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.445+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe080005080 con 0x7fe0a0145610 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.447+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe090004220 con 0x7fe0a0145610 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.447+0000 7fe0957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.447+0000 7fe0957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:59.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.509+0000 7fe0a5d9f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.509+0000 7fe0a5d9f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.509+0000 7fe0a5d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:19:59.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.509+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe080005330 con 0x7fe0a0145610 2024-03-29T11:19:59.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.509+0000 7fe0957fa640 1 -- 172.21.15.52:0/907610094 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fe0900043c0 con 0x7fe0a0145610 2024-03-29T11:19:59.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.510+0000 7fe0957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.510+0000 7fe0957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:59.487 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-29T11:19:59.487 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:59.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 10 librados: watch_flush enter 2024-03-29T11:19:59.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 10 librados: watch_flush exit 2024-03-29T11:19:59.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:19:59.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:19:59.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 10 monclient: shutdown 2024-03-29T11:19:59.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 >> v1:172.21.15.52:6789/0 conn(0x7fe0a0145610 legacy=0x7fe0a0136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 shutdown_connections 2024-03-29T11:19:59.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 >> 172.21.15.52:0/907610094 conn(0x7fe0a00726e0 msgr2=0x7fe0a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 shutdown_connections 2024-03-29T11:19:59.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 1 -- 172.21.15.52:0/907610094 wait complete. 2024-03-29T11:19:59.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-29T11:19:59.511+0000 7fe0a5d9f640 1 librados: shutdown 2024-03-29T11:19:59.501 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:59.501 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:59.507 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.530+0000 7fd29afb6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:59.524 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.548+0000 7f250a041040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:59.600 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.624+0000 7fd29afb6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:59.623 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.647+0000 7f250a041040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:59.756 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.779+0000 7fd29afb6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:59.783 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.807+0000 7f250a041040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:59.797 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.821+0000 7fd29afb6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:59.827 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:19:59.851+0000 7f250a041040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:59.851 INFO:tasks.ceph.osd.1.smithi052.stderr:2024-03-29T11:19:59.874+0000 7f49c2499640 -1 Falling back to public interface 2024-03-29T11:19:59.855 INFO:tasks.ceph.osd.0.smithi052.stderr:2024-03-29T11:19:59.878+0000 7fd47a40c640 -1 Falling back to public interface 2024-03-29T11:19:59.864 INFO:tasks.ceph.osd.3.smithi052.stderr:2024-03-29T11:19:59.887+0000 7fae4ab9a640 -1 Falling back to public interface 2024-03-29T11:19:59.871 INFO:tasks.ceph.osd.2.smithi052.stderr:2024-03-29T11:19:59.894+0000 7f0fdc981640 -1 Falling back to public interface 2024-03-29T11:19:59.901 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-03-29T11:19:59.924+0000 7f0f7ef2a640 -1 Falling back to public interface 2024-03-29T11:19:59.901 INFO:tasks.ceph.osd.6.smithi079.stderr:2024-03-29T11:19:59.924+0000 7f6d86f28640 -1 Falling back to public interface 2024-03-29T11:19:59.901 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-03-29T11:19:59.924+0000 7f0f7ded9640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T11:19:59.901 INFO:tasks.ceph.osd.6.smithi079.stderr:2024-03-29T11:19:59.924+0000 7f6d85ed7640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T11:19:59.902 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-03-29T11:19:59.924+0000 7f0f7ded9640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-29T11:19:59.905 INFO:tasks.ceph.osd.7.smithi079.stderr:2024-03-29T11:19:59.928+0000 7fa16b214640 -1 Falling back to public interface 2024-03-29T11:19:59.909 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-03-29T11:19:59.932+0000 7fe08240d640 -1 Falling back to public interface 2024-03-29T11:19:59.952 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:19:59.976+0000 7fd29afb6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:59.988 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:20:00.012+0000 7f250a041040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:20:00.307 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:00.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.420+0000 7faef0572640 10 monclient: get_monmap_and_config 2024-03-29T11:20:00.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.420+0000 7faef0572640 10 monclient: build_initial_monmap 2024-03-29T11:20:00.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.420+0000 7faef0572640 10 monclient: monmap: 2024-03-29T11:20:00.397 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:00.397 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:00.421722+0000 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:00.421722+0000 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:00.398 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.421+0000 7faef0572640 1 Processor -- start 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.421+0000 7faef0572640 1 -- start start 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.421+0000 7faef0572640 10 monclient: init 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient(hunting): picked mon.noname-b con 0x7faee8070170 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient(hunting): picked mon.noname-c con 0x7faee80713f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient(hunting): picked mon.noname-a con 0x7faee8070550 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee813dbd0 con 0x7faee8070550 2024-03-29T11:20:00.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee813ff70 con 0x7faee80713f0 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee8140250 con 0x7faee8070170 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faef0572640 10 monclient(hunting): authenticate will time out at 930.562541s 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeeeb0f640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7faee8070550 0x7faee814b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33448/0 (socket says 172.21.15.52:33448) 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeeeb0f640 1 -- 172.21.15.52:0/575849046 learned_addr learned my addr 172.21.15.52:0/575849046 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeee30e640 1 --1- 172.21.15.52:0/575849046 >> v1:172.21.15.52:6790/0 conn(0x7faee8070170 0x7faee813d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58952/0 (socket says 172.21.15.52:58952) 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433257146 0 0) 0x7faee8140250 con 0x7faee8070170 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _negotiate 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:20:00.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed4003520 con 0x7faee8070170 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359900280 0 0) 0x7faee0001490 con 0x7faee8070550 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _negotiate 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee8140250 con 0x7faee8070550 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088650515 0 0) 0x7faee813ff70 con 0x7faee80713f0 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _negotiate 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:20:00.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.422+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee0001490 con 0x7faee80713f0 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623074400 0 0) 0x7faee8140250 con 0x7faee8070550 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 >> v1:172.21.15.52:6790/0 conn(0x7faee8070170 legacy=0x7faee813d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 >> v1:172.21.15.79:6789/0 conn(0x7faee80713f0 legacy=0x7faee81406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: _finish_hunting 0 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 monclient: found mon.noname-a 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8140df0 con 0x7faee8070550 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: _finish_auth 0 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: _check_auth_tickets 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:00.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faee00019e0 con 0x7faee8070550 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: dump: 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:00.412 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee0001b40 con 0x7faee8070550 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 1 -- 172.21.15.52:0/575849046 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faee0001f90 con 0x7faee8070550 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 4 set_mon_vals no callback set 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faeed30c640 10 monclient: dump: 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.413 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 10 monclient: shutdown 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 1 -- 172.21.15.52:0/575849046 >> v1:172.21.15.52:6789/0 conn(0x7faee8070550 legacy=0x7faee814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 1 -- 172.21.15.52:0/575849046 shutdown_connections 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.423+0000 7faef0572640 1 -- 172.21.15.52:0/575849046 >> 172.21.15.52:0/575849046 conn(0x7faee8067610 msgr2=0x7faee80b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 1 -- 172.21.15.52:0/575849046 shutdown_connections 2024-03-29T11:20:00.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 1 -- 172.21.15.52:0/575849046 wait complete. 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 10 monclient: build_initial_monmap 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 10 monclient: monmap: 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:00.415 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 1 librados: starting msgr at 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 1 librados: starting objecter 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.424+0000 7faef0572640 1 Processor -- start 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 1 -- start start 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 1 librados: setting wanted keys 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 1 librados: calling monclient init 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient: init 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient(hunting): picked mon.noname-a con 0x7faee8070170 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient(hunting): picked mon.noname-b con 0x7faee8070550 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:00.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient(hunting): picked mon.noname-c con 0x7faee80713f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee8142e80 con 0x7faee8070170 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee8144240 con 0x7faee8070550 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee81430f0 con 0x7faee80713f0 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faef0572640 10 monclient(hunting): authenticate will time out at 930.565614s 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faeee30e640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7faee8070170 0x7faee8131b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33450/0 (socket says 172.21.15.52:33450) 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faeee30e640 1 -- 172.21.15.52:0/3065150368 learned_addr learned my addr 172.21.15.52:0/3065150368 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 client.?.objecter ms_handle_connect 0x7faee8070170 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 client.?.objecter ms_handle_connect 0x7faee80713f0 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 client.?.objecter ms_handle_connect 0x7faee8070550 2024-03-29T11:20:00.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008307521 0 0) 0x7faee81430f0 con 0x7faee80713f0 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc004eb0 con 0x7faee80713f0 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673475035 0 0) 0x7faee8142e80 con 0x7faee8070170 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:20:00.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee81430f0 con 0x7faee8070170 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183423828 0 0) 0x7faee8144240 con 0x7faee8070550 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.425+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee8142e80 con 0x7faee8070550 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031879740 0 0) 0x7faee8142e80 con 0x7faee8070550 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faee8144240 con 0x7faee8070550 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610586531 0 0) 0x7faecc004eb0 con 0x7faee80713f0 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faee8142e80 con 0x7faee80713f0 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1383819403 0 0) 0x7faee81430f0 con 0x7faee8070170 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faecc004eb0 con 0x7faee8070170 2024-03-29T11:20:00.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faed8001830 con 0x7faee8070550 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): dump: 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:00.420 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faee0004b60 con 0x7faee80713f0 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): dump: 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:00.421 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faee40017f0 con 0x7faee8070170 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient(hunting): dump: 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:00.422 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2935822438 0 0) 0x7faee8144240 con 0x7faee8070550 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 >> v1:172.21.15.52:6790/0 conn(0x7faee80713f0 legacy=0x7faee81dae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 >> v1:172.21.15.52:6789/0 conn(0x7faee8070170 legacy=0x7faee8131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 monclient: found mon.b 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee81433d0 con 0x7faee8070550 2024-03-29T11:20:00.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 5 monclient: authenticate success, global_id 4180 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 10 monclient: _renew_subs 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee806da30 con 0x7faee8070550 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faed8001990 con 0x7faee8070550 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faed80037f0 con 0x7faee8070550 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 monclient: dump: 2024-03-29T11:20:00.424 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedffff640 4 set_mon_vals no callback set 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 10 monclient: _renew_subs 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:00.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faee8132480 con 0x7faee8070550 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faef0572640 1 librados: init done 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faed8003ce0 con 0x7faee8070550 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7faed8004660 con 0x7faee8070550 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 10 client.4180.objecter ms_dispatch 0x7faee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.426+0000 7faedeffd640 20 client.4180.objecter dump_active .. 0 homeless 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.427+0000 7faef0572640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.427+0000 7faef0572640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.427+0000 7faef0572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.427+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee81438f0 con 0x7faee8070550 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.429+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faee81438f0 con 0x7faee8070550 2024-03-29T11:20:00.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.429+0000 7faedeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:00.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.429+0000 7faedeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:00.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.491+0000 7faef0572640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:00.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.491+0000 7faef0572640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:00.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.491+0000 7faef0572640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:00.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.491+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee8144090 con 0x7faee8070550 2024-03-29T11:20:00.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.492+0000 7faedeffd640 1 -- 172.21.15.52:0/3065150368 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7faed8003f90 con 0x7faee8070550 2024-03-29T11:20:00.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.492+0000 7faedeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:00.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.492+0000 7faedeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:00.469 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:00.469 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 10 librados: watch_flush enter 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 10 librados: watch_flush exit 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 20 client.4180.objecter shutdown clearing up homeless session... 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 10 client.4180.objecter successfully canceled tick 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 10 monclient: shutdown 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 >> v1:172.21.15.79:6789/0 conn(0x7faee8070550 legacy=0x7faee81da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 shutdown_connections 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 >> 172.21.15.52:0/3065150368 conn(0x7faee8067610 msgr2=0x7faee8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 shutdown_connections 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 1 -- 172.21.15.52:0/3065150368 wait complete. 2024-03-29T11:20:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:00.493+0000 7faef0572640 1 librados: shutdown 2024-03-29T11:20:00.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:00.764 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:20:00.787+0000 7fd29afb6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:00.809 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:20:00.832+0000 7f250a041040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:01.036 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:20:01.059+0000 7fd29afb6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:01.086 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:20:01.110+0000 7fd29afb6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:01.094 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:20:01.117+0000 7f250a041040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:01.146 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:20:01.170+0000 7f250a041040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:01.200 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:20:01.224+0000 7fd29afb6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:01.271 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-29T11:20:01.295+0000 7f250a041040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:02.215 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:20:02.238+0000 7fd23c417640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:20:02.215 INFO:tasks.ceph.mgr.x.smithi079.stderr:2024-03-29T11:20:02.239+0000 7fd23c417640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:20:03.292 INFO:tasks.ceph.osd.1.smithi052.stderr:2024-03-29T11:20:03.315+0000 7f49c2499640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:20:03.351 INFO:tasks.ceph.osd.0.smithi052.stderr:2024-03-29T11:20:03.375+0000 7fd47a40c640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:20:03.368 INFO:tasks.ceph.osd.7.smithi079.stderr:2024-03-29T11:20:03.392+0000 7fa16b214640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:20:03.385 INFO:tasks.ceph.osd.3.smithi052.stderr:2024-03-29T11:20:03.409+0000 7fae4ab9a640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:20:03.394 INFO:tasks.ceph.osd.2.smithi052.stderr:2024-03-29T11:20:03.418+0000 7f0fdc981640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:20:03.417 INFO:tasks.ceph.osd.6.smithi079.stderr:2024-03-29T11:20:03.440+0000 7f6d86f28640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:20:03.419 INFO:tasks.ceph.osd.5.smithi079.stderr:2024-03-29T11:20:03.443+0000 7fe08240d640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:20:03.433 INFO:tasks.ceph.osd.4.smithi079.stderr:2024-03-29T11:20:03.456+0000 7f0f7ef2a640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:20:07.286 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:07.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.473+0000 7fc3ee13e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:07.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.473+0000 7fc3ee13e640 10 monclient: build_initial_monmap 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.473+0000 7fc3ee13e640 10 monclient: monmap: 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:07.475603+0000 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:07.475603+0000 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:07.452 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:07.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.475+0000 7fc3ee13e640 1 Processor -- start 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.475+0000 7fc3ee13e640 1 -- start start 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.475+0000 7fc3ee13e640 10 monclient: init 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e81457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3e77fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc3e81457f0 0x7fc3e813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:53770/0 (socket says 172.21.15.52:53770) 2024-03-29T11:20:07.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3dffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc3e813f650 0x7fc3e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:48616/0 (socket says 172.21.15.52:48616) 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3e7fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc3e80a9d70 0x7fc3e814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:55898/0 (socket says 172.21.15.52:55898) 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e80a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80aaef0 con 0x7fc3e813f650 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8145bd0 con 0x7fc3e81457f0 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8145eb0 con 0x7fc3e80a9d70 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3ee13e640 10 monclient(hunting): authenticate will time out at 937.618011s 2024-03-29T11:20:07.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.476+0000 7fc3e77fe640 1 -- 172.21.15.52:0/2674410456 learned_addr learned my addr 172.21.15.52:0/2674410456 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.477+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175625027 0 0) 0x7fc3e80aaef0 con 0x7fc3e813f650 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4003520 con 0x7fc3e813f650 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083986645 0 0) 0x7fc3e8145eb0 con 0x7fc3e80a9d70 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:20:07.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e80aaef0 con 0x7fc3e80a9d70 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584445980 0 0) 0x7fc3e8145bd0 con 0x7fc3e81457f0 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient(hunting): my global_id is 4219 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8145eb0 con 0x7fc3e81457f0 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843090449 0 0) 0x7fc3e8145eb0 con 0x7fc3e81457f0 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 >> v1:172.21.15.52:6790/0 conn(0x7fc3e80a9d70 legacy=0x7fc3e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 >> v1:172.21.15.52:6789/0 conn(0x7fc3e813f650 legacy=0x7fc3e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:07.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8146340 con 0x7fc3e81457f0 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc3d8001530 con 0x7fc3e81457f0 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: dump: 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:07.459 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3ee13e640 5 monclient: authenticate success, global_id 4219 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3ee13e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3d8001690 con 0x7fc3e81457f0 2024-03-29T11:20:07.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 1 -- 172.21.15.52:0/2674410456 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc3d8001a60 con 0x7fc3e81457f0 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.478+0000 7fc3e6ffd640 10 monclient: dump: 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:07.461 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:07.462 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:07.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.479+0000 7fc3ee13e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:07.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.479+0000 7fc3ee13e640 4 set_mon_vals no callback set 2024-03-29T11:20:07.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 10 monclient: shutdown 2024-03-29T11:20:07.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:07.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2674410456 >> v1:172.21.15.79:6789/0 conn(0x7fc3e81457f0 legacy=0x7fc3e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2674410456 shutdown_connections 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2674410456 >> 172.21.15.52:0/2674410456 conn(0x7fc3e8066a00 msgr2=0x7fc3e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2674410456 shutdown_connections 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.480+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2674410456 wait complete. 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 10 monclient: build_initial_monmap 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 10 monclient: monmap: 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:07.470 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 1 librados: starting msgr at 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 1 librados: starting objecter 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 1 Processor -- start 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 1 -- start start 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 1 librados: setting wanted keys 2024-03-29T11:20:07.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 1 librados: calling monclient init 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.482+0000 7fc3ee13e640 10 monclient: init 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e81457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8071d40 con 0x7fc3e80a9d70 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3dffff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc3e813f650 0x7fc3e8071630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56062/0 (socket says 172.21.15.52:56062) 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3dffff640 1 -- 172.21.15.52:0/2095916533 learned_addr learned my addr 172.21.15.52:0/2095916533 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80ac080 con 0x7fc3e81457f0 2024-03-29T11:20:07.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80ac360 con 0x7fc3e813f650 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3ee13e640 10 monclient(hunting): authenticate will time out at 937.625796s 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3e4ff9640 10 client.?.objecter ms_handle_connect 0x7fc3e813f650 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3e4ff9640 10 client.?.objecter ms_handle_connect 0x7fc3e81457f0 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.484+0000 7fc3e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438341182 0 0) 0x7fc3e80ac080 con 0x7fc3e81457f0 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): my global_id is 4222 2024-03-29T11:20:07.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8004eb0 con 0x7fc3e81457f0 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776915656 0 0) 0x7fc3e80ac360 con 0x7fc3e813f650 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.485+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e80ac080 con 0x7fc3e813f650 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 client.?.objecter ms_handle_connect 0x7fc3e80a9d70 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:07.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184938038 0 0) 0x7fc3c8004eb0 con 0x7fc3e81457f0 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e80ac360 con 0x7fc3e81457f0 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc3e0001c30 con 0x7fc3e81457f0 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:07.475 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769464958 0 0) 0x7fc3e8071d40 con 0x7fc3e80a9d70 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8004eb0 con 0x7fc3e80a9d70 2024-03-29T11:20:07.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1255975294 0 0) 0x7fc3e80ac360 con 0x7fc3e81457f0 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.486+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 >> v1:172.21.15.52:6790/0 conn(0x7fc3e813f650 legacy=0x7fc3e8071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 >> v1:172.21.15.52:6789/0 conn(0x7fc3e80a9d70 legacy=0x7fc3e8070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 1 monclient: found mon.b 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80ac640 con 0x7fc3e81457f0 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 5 monclient: authenticate success, global_id 4222 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 10 monclient: _renew_subs 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:07.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e80b1680 con 0x7fc3e81457f0 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 10 monclient: _renew_subs 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e80b1230 con 0x7fc3e81457f0 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3ee13e640 1 librados: init done 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3e0004b60 con 0x7fc3e81457f0 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc3e0004f70 con 0x7fc3e81457f0 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e67fc640 4 set_mon_vals no callback set 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:07.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.487+0000 7fc3e4ff9640 10 monclient: dump: 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:07.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.490+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fc3e00050d0 con 0x7fc3e81457f0 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.490+0000 7fc3ee13e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.490+0000 7fc3ee13e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3ee13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc005080 con 0x7fc3e81457f0 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3e77fe640 10 monclient: get_auth_request con 0x7fc3c806dbe0 auth_method 0 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fc3e0084d00 con 0x7fc3e81457f0 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3e4ff9640 10 client.4222.objecter ms_dispatch 0x7fc3e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:07.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3e4ff9640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:07.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.491+0000 7fc3e4ff9640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:07.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.492+0000 7fc3e4ff9640 20 client.4222.objecter dump_active .. 0 homeless 2024-03-29T11:20:07.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.492+0000 7fc3e4ff9640 10 client.4222.objecter ms_handle_connect 0x7fc3c806dbe0 2024-03-29T11:20:07.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.498+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3e00505e0 con 0x7fc3e81457f0 2024-03-29T11:20:07.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.498+0000 7fc3e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:07.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.498+0000 7fc3e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:07.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.904+0000 7fc3ee13e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:07.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.904+0000 7fc3ee13e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:07.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.904+0000 7fc3ee13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:07.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.904+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3cc005330 con 0x7fc3e81457f0 2024-03-29T11:20:07.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.905+0000 7fc3e4ff9640 1 -- 172.21.15.52:0/2095916533 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fc3e0052510 con 0x7fc3e81457f0 2024-03-29T11:20:07.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.905+0000 7fc3e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:07.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.905+0000 7fc3e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:07.883 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:07.883 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:07.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 10 librados: watch_flush enter 2024-03-29T11:20:07.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 10 librados: watch_flush exit 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 20 client.4222.objecter shutdown clearing up homeless session... 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 10 client.4222.objecter successfully canceled tick 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 >> v1:172.21.15.79:6816/407480584 conn(0x7fc3c806dbe0 legacy=0x7fc3c80700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 10 monclient: shutdown 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.909+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 >> v1:172.21.15.79:6789/0 conn(0x7fc3e81457f0 legacy=0x7fc3e81e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.910+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 shutdown_connections 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.910+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 >> 172.21.15.52:0/2095916533 conn(0x7fc3e8066a00 msgr2=0x7fc3e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.910+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 shutdown_connections 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.910+0000 7fc3ee13e640 1 -- 172.21.15.52:0/2095916533 wait complete. 2024-03-29T11:20:07.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:07.910+0000 7fc3ee13e640 1 librados: shutdown 2024-03-29T11:20:07.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:14.709 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.909+0000 7fe0f50cc640 10 monclient: get_monmap_and_config 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.909+0000 7fe0f50cc640 10 monclient: build_initial_monmap 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.909+0000 7fe0f50cc640 10 monclient: monmap: 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:14.910209+0000 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:14.910209+0000 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:14.886 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.887 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:14.887 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:14.887 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:14.887 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.910+0000 7fe0f50cc640 1 Processor -- start 2024-03-29T11:20:14.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.910+0000 7fe0f50cc640 1 -- start start 2024-03-29T11:20:14.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.910+0000 7fe0f50cc640 10 monclient: init 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.911+0000 7fe0f50cc640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.911+0000 7fe0f50cc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.911+0000 7fe0f50cc640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e80389e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.911+0000 7fe0f50cc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e8038dc0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f50cc640 10 monclient(hunting): picked mon.noname-b con 0x7fe0e8030660 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f50cc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80c54c0 con 0x7fe0e8038dc0 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f50cc640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e8036430 con 0x7fe0e80389e0 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f50cc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e8039840 con 0x7fe0e8030660 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f50cc640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:14.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f50cc640 10 monclient(hunting): authenticate will time out at 945.052273s 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f48cb640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe0e8030660 0x7fe0e80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:54812/0 (socket says 172.21.15.52:54812) 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0f48cb640 1 -- 172.21.15.52:0/3501715852 learned_addr learned my addr 172.21.15.52:0/3501715852 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101117530 0 0) 0x7fe0e8039840 con 0x7fe0e8030660 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0eeffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.912+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e0003520 con 0x7fe0e8030660 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748432354 0 0) 0x7fe0e80c54c0 con 0x7fe0e8038dc0 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e8039840 con 0x7fe0e8038dc0 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931461099 0 0) 0x7fe0e0003520 con 0x7fe0e8030660 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 >> v1:172.21.15.79:6789/0 conn(0x7fe0e80389e0 legacy=0x7fe0e8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 >> v1:172.21.15.52:6789/0 conn(0x7fe0e8038dc0 legacy=0x7fe0e80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 monclient: found mon.noname-b 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e8035690 con 0x7fe0e8030660 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe0f004f730 con 0x7fe0e8030660 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0f50cc640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:20:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0f50cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0eeffd640 10 monclient: dump: 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.913+0000 7fe0f50cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0f00008d0 con 0x7fe0e8030660 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0eeffd640 1 -- 172.21.15.52:0/3501715852 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe0f00758b0 con 0x7fe0e8030660 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0eeffd640 10 monclient: dump: 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.897 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0f50cc640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.914+0000 7fe0f50cc640 4 set_mon_vals no callback set 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.915+0000 7fe0f50cc640 10 monclient: shutdown 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.915+0000 7fe0f50cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.915+0000 7fe0f50cc640 1 -- 172.21.15.52:0/3501715852 >> v1:172.21.15.52:6790/0 conn(0x7fe0e8030660 legacy=0x7fe0e80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.915+0000 7fe0f50cc640 1 -- 172.21.15.52:0/3501715852 shutdown_connections 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.915+0000 7fe0f50cc640 1 -- 172.21.15.52:0/3501715852 >> 172.21.15.52:0/3501715852 conn(0x7fe0e8005c00 msgr2=0x7fe0e8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:14.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.916+0000 7fe0f50cc640 1 -- 172.21.15.52:0/3501715852 shutdown_connections 2024-03-29T11:20:14.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.916+0000 7fe0f50cc640 1 -- 172.21.15.52:0/3501715852 wait complete. 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.917+0000 7fe0f50cc640 10 monclient: build_initial_monmap 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.917+0000 7fe0f50cc640 10 monclient: monmap: 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:14.906 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.918+0000 7fe0f50cc640 1 librados: starting msgr at 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.918+0000 7fe0f50cc640 1 librados: starting objecter 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.918+0000 7fe0f50cc640 1 Processor -- start 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.919+0000 7fe0f50cc640 1 -- start start 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.919+0000 7fe0f50cc640 1 librados: setting wanted keys 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.919+0000 7fe0f50cc640 1 librados: calling monclient init 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.919+0000 7fe0f50cc640 10 monclient: init 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient(hunting): picked mon.noname-b con 0x7fe0e8030660 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e80389e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e800a540 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80c3870 con 0x7fe0e800a540 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e800d740 con 0x7fe0e8030660 2024-03-29T11:20:14.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e800b110 con 0x7fe0e80389e0 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0f50cc640 10 monclient(hunting): authenticate will time out at 945.060661s 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0ef7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe0e80389e0 0x7fe0e800d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:55900/0 (socket says 172.21.15.52:55900) 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0ef7fe640 1 -- 172.21.15.52:0/685260578 learned_addr learned my addr 172.21.15.52:0/685260578 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0ecff9640 10 client.?.objecter ms_handle_connect 0x7fe0e80389e0 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.920+0000 7fe0ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 client.?.objecter ms_handle_connect 0x7fe0e800a540 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818705742 0 0) 0x7fe0e800b110 con 0x7fe0e80389e0 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4004eb0 con 0x7fe0e80389e0 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462300345 0 0) 0x7fe0e80c3870 con 0x7fe0e800a540 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e800b110 con 0x7fe0e800a540 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2349854977 0 0) 0x7fe0d4004eb0 con 0x7fe0e80389e0 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0e80c3870 con 0x7fe0e80389e0 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe0dc001930 con 0x7fe0e80389e0 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): dump: 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.921+0000 7fe0ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:14.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712184256 0 0) 0x7fe0e800b110 con 0x7fe0e800a540 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d4004eb0 con 0x7fe0e800a540 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 client.?.objecter ms_handle_connect 0x7fe0e8030660 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe0f004f400 con 0x7fe0e800a540 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient(hunting): dump: 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:14.911 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 794537635 0 0) 0x7fe0e80c3870 con 0x7fe0e80389e0 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 >> v1:172.21.15.79:6789/0 conn(0x7fe0e8030660 legacy=0x7fe0e800b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 >> v1:172.21.15.52:6789/0 conn(0x7fe0e800a540 legacy=0x7fe0e816b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 monclient: found mon.c 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e800d970 con 0x7fe0e80389e0 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 10 monclient: _renew_subs 2024-03-29T11:20:14.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e80c3a50 con 0x7fe0e80389e0 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 10 monclient: _renew_subs 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e816ca90 con 0x7fe0e80389e0 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.922+0000 7fe0f50cc640 1 librados: init done 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0dc001cf0 con 0x7fe0e80389e0 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe0dc0038a0 con 0x7fe0e80389e0 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ecff9640 10 monclient: dump: 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:14.913 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.923+0000 7fe0ee7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.924+0000 7fe0f50cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.925+0000 7fe0f50cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.925+0000 7fe0f50cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:14.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.925+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0005080 con 0x7fe0e80389e0 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.925+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fe0dc003a00 con 0x7fe0e80389e0 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.926+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fe0dc084a80 con 0x7fe0e80389e0 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.926+0000 7fe0ecff9640 10 client.4250.objecter ms_dispatch 0x7fe0f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.926+0000 7fe0ecff9640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.926+0000 7fe0ecff9640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.926+0000 7fe0ecff9640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.926+0000 7fe0effff640 10 monclient: get_auth_request con 0x7fe0d406dd90 auth_method 0 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.927+0000 7fe0ecff9640 10 client.4250.objecter ms_handle_connect 0x7fe0d406dd90 2024-03-29T11:20:14.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.937+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0dc0503c0 con 0x7fe0e80389e0 2024-03-29T11:20:14.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.937+0000 7fe0ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:14.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:14.937+0000 7fe0ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:15.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.264+0000 7fe0f50cc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:15.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.264+0000 7fe0f50cc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:15.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.264+0000 7fe0f50cc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:15.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.264+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0d0005330 con 0x7fe0e80389e0 2024-03-29T11:20:15.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.265+0000 7fe0ecff9640 1 -- 172.21.15.52:0/685260578 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fe0dc0522f0 con 0x7fe0e80389e0 2024-03-29T11:20:15.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.265+0000 7fe0ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:15.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.265+0000 7fe0ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:15.242 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:15.242 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 10 librados: watch_flush enter 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 10 librados: watch_flush exit 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 >> v1:172.21.15.79:6816/407480584 conn(0x7fe0d406dd90 legacy=0x7fe0d4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 10 monclient: shutdown 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 >> v1:172.21.15.52:6790/0 conn(0x7fe0e80389e0 legacy=0x7fe0e800d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 shutdown_connections 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 >> 172.21.15.52:0/685260578 conn(0x7fe0e8005c00 msgr2=0x7fe0e8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:15.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 shutdown_connections 2024-03-29T11:20:15.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 -- 172.21.15.52:0/685260578 wait complete. 2024-03-29T11:20:15.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:15.268+0000 7fe0f50cc640 1 librados: shutdown 2024-03-29T11:20:15.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:22.065 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.238+0000 7f6ff0167640 10 monclient: get_monmap_and_config 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.238+0000 7f6ff0167640 10 monclient: build_initial_monmap 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.238+0000 7f6ff0167640 10 monclient: monmap: 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:22.239396+0000 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:22.239396+0000 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:22.215 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.242+0000 7f6ff0167640 1 Processor -- start 2024-03-29T11:20:22.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.243+0000 7f6ff0167640 1 -- start start 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.243+0000 7f6ff0167640 10 monclient: init 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe81422b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe8142690 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe80b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe8142c80 con 0x7f6fe81422b0 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe80aae40 con 0x7f6fe80b17d0 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe813fbd0 con 0x7f6fe8142690 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:22.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.244+0000 7f6ff0167640 10 monclient(hunting): authenticate will time out at 952.384688s 2024-03-29T11:20:22.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.247+0000 7f6fed702640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f6fe8142690 0x7f6fe80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:34065/0 (socket says 172.21.15.52:34065) 2024-03-29T11:20:22.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.247+0000 7f6fed702640 1 -- 172.21.15.52:0/2374122565 learned_addr learned my addr 172.21.15.52:0/2374122565 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:22.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663942019 0 0) 0x7f6fe813fbd0 con 0x7f6fe8142690 2024-03-29T11:20:22.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): _negotiate 2024-03-29T11:20:22.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:22.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:22.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:20:22.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4003520 con 0x7f6fe8142690 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480558422 0 0) 0x7f6fe8142c80 con 0x7f6fe81422b0 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): _negotiate 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe813fbd0 con 0x7f6fe81422b0 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 42456486 0 0) 0x7f6fe813fbd0 con 0x7f6fe81422b0 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 >> v1:172.21.15.52:6790/0 conn(0x7f6fe8142690 legacy=0x7f6fe80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 >> v1:172.21.15.79:6789/0 conn(0x7f6fe80b17d0 legacy=0x7f6fe80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:22.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: _finish_hunting 0 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 monclient: found mon.noname-a 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8151940 con 0x7f6fe81422b0 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: _finish_auth 0 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: _check_auth_tickets 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6fd8001670 con 0x7f6fe81422b0 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.248+0000 7f6fecf01640 10 monclient: dump: 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:22.227 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6ff0167640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6ff0167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fd80017d0 con 0x7f6fe81422b0 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6fecf01640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6fecf01640 1 -- 172.21.15.52:0/2374122565 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6fd8001b60 con 0x7f6fe81422b0 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6fecf01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6fecf01640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6fecf01640 10 monclient: dump: 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:22.229 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6ff0167640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.249+0000 7f6ff0167640 4 set_mon_vals no callback set 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.250+0000 7f6ff0167640 10 monclient: shutdown 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.250+0000 7f6ff0167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.250+0000 7f6ff0167640 1 -- 172.21.15.52:0/2374122565 >> v1:172.21.15.52:6789/0 conn(0x7f6fe81422b0 legacy=0x7f6fe813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.250+0000 7f6ff0167640 1 -- 172.21.15.52:0/2374122565 shutdown_connections 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.250+0000 7f6ff0167640 1 -- 172.21.15.52:0/2374122565 >> 172.21.15.52:0/2374122565 conn(0x7f6fe8066a00 msgr2=0x7f6fe8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.251+0000 7f6ff0167640 1 -- 172.21.15.52:0/2374122565 shutdown_connections 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.251+0000 7f6ff0167640 1 -- 172.21.15.52:0/2374122565 wait complete. 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.252+0000 7f6ff0167640 10 monclient: build_initial_monmap 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.252+0000 7f6ff0167640 10 monclient: monmap: 2024-03-29T11:20:22.231 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.252+0000 7f6ff0167640 1 librados: starting msgr at 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.252+0000 7f6ff0167640 1 librados: starting objecter 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.252+0000 7f6ff0167640 1 Processor -- start 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.253+0000 7f6ff0167640 1 -- start start 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.253+0000 7f6ff0167640 1 librados: setting wanted keys 2024-03-29T11:20:22.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.253+0000 7f6ff0167640 1 librados: calling monclient init 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.253+0000 7f6ff0167640 10 monclient: init 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.253+0000 7f6ff0167640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.253+0000 7f6ff0167640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.256+0000 7f6ff0167640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe80b17d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.256+0000 7f6fedf03640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6fe80b17d0 0x7f6fe80ad030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:36711/0 (socket says 172.21.15.52:36711) 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.256+0000 7f6fedf03640 1 -- 172.21.15.52:0/2364960387 learned_addr learned my addr 172.21.15.52:0/2364960387 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:22.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.256+0000 7f6ff0167640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe81422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:22.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.256+0000 7f6ff0167640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe8142690 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:22.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.257+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe806ff70 con 0x7f6fe80b17d0 2024-03-29T11:20:22.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.257+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe81d7600 con 0x7f6fe81422b0 2024-03-29T11:20:22.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.257+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe80ad740 con 0x7f6fe8142690 2024-03-29T11:20:22.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.257+0000 7f6ff0167640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.257+0000 7f6ff0167640 10 monclient(hunting): authenticate will time out at 952.397411s 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.256+0000 7f6fde7fc640 10 client.?.objecter ms_handle_connect 0x7f6fe80b17d0 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 client.?.objecter ms_handle_connect 0x7f6fe81422b0 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152059390 0 0) 0x7f6fe81d7600 con 0x7f6fe81422b0 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:20:22.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc004eb0 con 0x7f6fe81422b0 2024-03-29T11:20:22.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399684668 0 0) 0x7f6fe806ff70 con 0x7f6fe80b17d0 2024-03-29T11:20:22.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:22.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe81d7600 con 0x7f6fe80b17d0 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 client.?.objecter ms_handle_connect 0x7f6fe8142690 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.260+0000 7f6fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941187701 0 0) 0x7f6fe81d7600 con 0x7f6fe80b17d0 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fcc005490 con 0x7f6fe80b17d0 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6fd8001ba0 con 0x7f6fe80b17d0 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.241 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375598602 0 0) 0x7f6fe80ad740 con 0x7f6fe8142690 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:20:22.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe81d7600 con 0x7f6fe8142690 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1061147377 0 0) 0x7f6fcc004eb0 con 0x7f6fe81422b0 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe80ad740 con 0x7f6fe81422b0 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.261+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 648838800 0 0) 0x7f6fe81d7600 con 0x7f6fe8142690 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fcc004eb0 con 0x7f6fe8142690 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6fe0001770 con 0x7f6fe81422b0 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.243 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6fe4001810 con 0x7f6fe8142690 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.244 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655544344 0 0) 0x7f6fcc005490 con 0x7f6fe80b17d0 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 >> v1:172.21.15.52:6790/0 conn(0x7f6fe8142690 legacy=0x7f6fe81d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 >> v1:172.21.15.79:6789/0 conn(0x7f6fe81422b0 legacy=0x7f6fe81d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 monclient: found mon.a 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8070f20 con 0x7f6fe80b17d0 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:22.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.262+0000 7f6fde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 10 monclient: _renew_subs 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fd80038a0 con 0x7f6fe80b17d0 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe8071110 con 0x7f6fe80b17d0 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 10 monclient: _renew_subs 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe80ab990 con 0x7f6fe80b17d0 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6ff0167640 1 librados: init done 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6fd8003cb0 con 0x7f6fe80b17d0 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:22.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fde7fc640 10 monclient: dump: 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:22.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.263+0000 7f6fdf7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:22.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.265+0000 7f6ff0167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:22.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.265+0000 7f6ff0167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.265+0000 7f6ff0167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.265+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe80acd60 con 0x7f6fe80b17d0 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.265+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f6fd8003e90 con 0x7f6fe80b17d0 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.266+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f6fd8084bc0 con 0x7f6fe80b17d0 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.266+0000 7f6fde7fc640 10 client.4257.objecter ms_dispatch 0x7f6fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.266+0000 7f6fde7fc640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.266+0000 7f6fde7fc640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.266+0000 7f6fde7fc640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.266+0000 7f6fed702640 10 monclient: get_auth_request con 0x7f6fcc06e340 auth_method 0 2024-03-29T11:20:22.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.267+0000 7f6fde7fc640 10 client.4257.objecter ms_handle_connect 0x7f6fcc06e340 2024-03-29T11:20:22.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.274+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fd8050500 con 0x7f6fe80b17d0 2024-03-29T11:20:22.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.274+0000 7f6fde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:22.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.274+0000 7f6fde7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:22.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.609+0000 7f6ff0167640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:22.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.609+0000 7f6ff0167640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:22.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.609+0000 7f6ff0167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:22.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.609+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fe8075f10 con 0x7f6fe80b17d0 2024-03-29T11:20:22.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.610+0000 7f6fde7fc640 1 -- 172.21.15.52:0/2364960387 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f6fd8050500 con 0x7f6fe80b17d0 2024-03-29T11:20:22.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.610+0000 7f6fde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:22.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.610+0000 7f6fde7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:22.587 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:22.588 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:22.590 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.614+0000 7f6ff0167640 10 librados: watch_flush enter 2024-03-29T11:20:22.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 10 librados: watch_flush exit 2024-03-29T11:20:22.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T11:20:22.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 10 client.4257.objecter successfully canceled tick 2024-03-29T11:20:22.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 >> v1:172.21.15.79:6816/407480584 conn(0x7f6fcc06e340 legacy=0x7f6fcc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:22.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 10 monclient: shutdown 2024-03-29T11:20:22.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:22.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.615+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 >> v1:172.21.15.52:6789/0 conn(0x7f6fe80b17d0 legacy=0x7f6fe80ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:22.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.616+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 shutdown_connections 2024-03-29T11:20:22.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.616+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 >> 172.21.15.52:0/2364960387 conn(0x7f6fe8066a00 msgr2=0x7f6fe81418c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:22.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.617+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 shutdown_connections 2024-03-29T11:20:22.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.617+0000 7f6ff0167640 1 -- 172.21.15.52:0/2364960387 wait complete. 2024-03-29T11:20:22.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:22.617+0000 7f6ff0167640 1 librados: shutdown 2024-03-29T11:20:22.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:29.418 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:29.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.590+0000 7f425f1ed640 10 monclient: get_monmap_and_config 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.590+0000 7f425f1ed640 10 monclient: build_initial_monmap 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.590+0000 7f425f1ed640 10 monclient: monmap: 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:29.591054+0000 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:29.591054+0000 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:29.567 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 1 Processor -- start 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 1 -- start start 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient: init 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient(hunting): picked mon.noname-a con 0x7f42581457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient(hunting): picked mon.noname-c con 0x7f425813f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:29.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient(hunting): picked mon.noname-b con 0x7f42580a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42580aaef0 con 0x7f42581457f0 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4258145bd0 con 0x7f425813f650 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4258145eb0 con 0x7f42580a9d70 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.591+0000 7f425f1ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.592+0000 7f425f1ed640 10 monclient(hunting): authenticate will time out at 959.732072s 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.592+0000 7f4257fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f425813f650 0x7f42580b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:37806/0 (socket says 172.21.15.52:37806) 2024-03-29T11:20:29.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.592+0000 7f4257fff640 1 -- 172.21.15.52:0/767275724 learned_addr learned my addr 172.21.15.52:0/767275724 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:29.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012504442 0 0) 0x7f4258145bd0 con 0x7f425813f650 2024-03-29T11:20:29.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:29.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:29.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:29.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:20:29.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248003520 con 0x7f425813f650 2024-03-29T11:20:29.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271220401 0 0) 0x7f4258145eb0 con 0x7f42580a9d70 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.595+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4258145bd0 con 0x7f42580a9d70 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663754163 0 0) 0x7f4248003520 con 0x7f425813f650 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 >> v1:172.21.15.52:6790/0 conn(0x7f42580a9d70 legacy=0x7f425814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 >> v1:172.21.15.52:6789/0 conn(0x7f42581457f0 legacy=0x7f425813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 monclient: found mon.noname-c 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:29.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258146340 con 0x7f425813f650 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4244001690 con 0x7f425813f650 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: dump: 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.575 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f425f1ed640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f425f1ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42440017f0 con 0x7f425813f650 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 1 -- 172.21.15.52:0/767275724 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4244001c00 con 0x7f425813f650 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f42577fe640 10 monclient: dump: 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:29.576 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f425f1ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.596+0000 7f425f1ed640 4 set_mon_vals no callback set 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.597+0000 7f425f1ed640 10 monclient: shutdown 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.597+0000 7f425f1ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.597+0000 7f425f1ed640 1 -- 172.21.15.52:0/767275724 >> v1:172.21.15.79:6789/0 conn(0x7f425813f650 legacy=0x7f42580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.597+0000 7f425f1ed640 1 -- 172.21.15.52:0/767275724 shutdown_connections 2024-03-29T11:20:29.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.597+0000 7f425f1ed640 1 -- 172.21.15.52:0/767275724 >> 172.21.15.52:0/767275724 conn(0x7f4258066a00 msgr2=0x7f4258066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.598+0000 7f425f1ed640 1 -- 172.21.15.52:0/767275724 shutdown_connections 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.598+0000 7f425f1ed640 1 -- 172.21.15.52:0/767275724 wait complete. 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.599+0000 7f425f1ed640 10 monclient: build_initial_monmap 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.599+0000 7f425f1ed640 10 monclient: monmap: 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:29.579 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 1 librados: starting msgr at 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 1 librados: starting objecter 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 1 Processor -- start 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 1 -- start start 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 1 librados: setting wanted keys 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 1 librados: calling monclient init 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.600+0000 7f425f1ed640 10 monclient: init 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient(hunting): picked mon.noname-b con 0x7f42580a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:29.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient(hunting): picked mon.noname-c con 0x7f425813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient(hunting): picked mon.noname-a con 0x7f42581457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42581cf1e0 con 0x7f42581457f0 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42581cc7d0 con 0x7f42580a9d70 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42581ce350 con 0x7f425813f650 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.601+0000 7f425f1ed640 10 monclient(hunting): authenticate will time out at 959.742025s 2024-03-29T11:20:29.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.603+0000 7f425cf89640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f42580a9d70 0x7f425813b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:38156/0 (socket says 172.21.15.52:38156) 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.603+0000 7f425cf89640 1 -- 172.21.15.52:0/191972520 learned_addr learned my addr 172.21.15.52:0/191972520 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 10 client.?.objecter ms_handle_connect 0x7f42580a9d70 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546484152 0 0) 0x7f42581cc7d0 con 0x7f42580a9d70 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:20:29.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.604+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423c004eb0 con 0x7f42580a9d70 2024-03-29T11:20:29.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.605+0000 7f42557fa640 10 client.?.objecter ms_handle_connect 0x7f425813f650 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.605+0000 7f42557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377640019 0 0) 0x7f42581ce350 con 0x7f425813f650 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42581cc7d0 con 0x7f425813f650 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 client.?.objecter ms_handle_connect 0x7f42581457f0 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229320156 0 0) 0x7f4258145bd0 con 0x7f42581457f0 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:29.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42581ce350 con 0x7f42581457f0 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840338052 0 0) 0x7f42581cc7d0 con 0x7f425813f650 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4258145bd0 con 0x7f425813f650 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4244001b00 con 0x7f425813f650 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.606+0000 7f42557fa640 10 monclient(hunting): dump: 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.586 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2410369577 0 0) 0x7f42581ce350 con 0x7f42581457f0 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42581cc7d0 con 0x7f42581457f0 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4250001b10 con 0x7f42581457f0 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 10 monclient(hunting): dump: 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.587 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3191819755 0 0) 0x7f4258145bd0 con 0x7f425813f650 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 >> v1:172.21.15.79:6789/0 conn(0x7f42580a9d70 legacy=0x7f425813b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.607+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 >> v1:172.21.15.52:6789/0 conn(0x7f42581457f0 legacy=0x7f42581e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 1 monclient: found mon.c 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42581ccab0 con 0x7f425813f650 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:29.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4244004a90 con 0x7f425813f650 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4244004da0 con 0x7f425813f650 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.608+0000 7f42557fa640 10 monclient: dump: 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:29.589 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:29.590 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:29.590 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:29.590 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:29.590 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:29.590 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:29.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.609+0000 7f4256ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:29.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:20:29.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 10 monclient: _renew_subs 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42581ccd90 con 0x7f425813f650 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 10 monclient: _renew_subs 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42581cecc0 con 0x7f425813f650 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.610+0000 7f425f1ed640 1 librados: init done 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.612+0000 7f425f1ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.612+0000 7f425f1ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.612+0000 7f425f1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.612+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4258075f10 con 0x7f425813f650 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.613+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f424401ad70 con 0x7f425813f650 2024-03-29T11:20:29.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.614+0000 7f425cf89640 10 monclient: get_auth_request con 0x7f423c06dfd0 auth_method 0 2024-03-29T11:20:29.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.614+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f4244084a80 con 0x7f425813f650 2024-03-29T11:20:29.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.614+0000 7f42557fa640 10 client.4262.objecter ms_dispatch 0x7f4258000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:29.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.614+0000 7f42557fa640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:29.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.614+0000 7f42557fa640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:29.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.614+0000 7f42557fa640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:20:29.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.615+0000 7f42557fa640 10 client.4262.objecter ms_handle_connect 0x7f423c06dfd0 2024-03-29T11:20:29.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.623+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4244050190 con 0x7f425813f650 2024-03-29T11:20:29.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.623+0000 7f42557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:29.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.623+0000 7f42557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:29.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.962+0000 7f425f1ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:29.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.962+0000 7f425f1ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:29.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.962+0000 7f425f1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:29.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.962+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42580b3680 con 0x7f425813f650 2024-03-29T11:20:29.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.963+0000 7f42557fa640 1 -- 172.21.15.52:0/191972520 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f4244052290 con 0x7f425813f650 2024-03-29T11:20:29.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.963+0000 7f42557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:29.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.963+0000 7f42557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:29.940 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:29.940 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:29.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 10 librados: watch_flush enter 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 10 librados: watch_flush exit 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 >> v1:172.21.15.79:6816/407480584 conn(0x7f423c06dfd0 legacy=0x7f423c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 10 monclient: shutdown 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:29.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.968+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 >> v1:172.21.15.52:6790/0 conn(0x7f425813f650 legacy=0x7f4258139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.971+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 shutdown_connections 2024-03-29T11:20:29.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.971+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 >> 172.21.15.52:0/191972520 conn(0x7f4258066a00 msgr2=0x7f425813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:29.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.971+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 shutdown_connections 2024-03-29T11:20:29.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.971+0000 7f425f1ed640 1 -- 172.21.15.52:0/191972520 wait complete. 2024-03-29T11:20:29.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:29.971+0000 7f425f1ed640 1 librados: shutdown 2024-03-29T11:20:29.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:36.772 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:36.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.952+0000 7f56613e2640 10 monclient: get_monmap_and_config 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.952+0000 7f56613e2640 10 monclient: build_initial_monmap 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.952+0000 7f56613e2640 10 monclient: monmap: 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:36.953181+0000 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:36.953181+0000 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:36.929 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:36.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.953+0000 7f56613e2640 1 Processor -- start 2024-03-29T11:20:36.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.956+0000 7f56613e2640 1 -- start start 2024-03-29T11:20:36.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.956+0000 7f56613e2640 10 monclient: init 2024-03-29T11:20:36.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:36.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:36.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient(hunting): picked mon.noname-b con 0x7f565c0b0770 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:36.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient(hunting): picked mon.noname-a con 0x7f565c14a700 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:36.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient(hunting): picked mon.noname-c con 0x7f565c0ac5a0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:36.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0aba70 con 0x7f565c14a700 2024-03-29T11:20:36.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0aecb0 con 0x7f565c0ac5a0 2024-03-29T11:20:36.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0af150 con 0x7f565c0b0770 2024-03-29T11:20:36.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:36.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.957+0000 7f56613e2640 10 monclient(hunting): authenticate will time out at 967.097404s 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.962+0000 7f565b7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f565c0ac5a0 0x7f565c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:59789/0 (socket says 172.21.15.52:59789) 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.962+0000 7f565b7fe640 1 -- 172.21.15.52:0/3206337736 learned_addr learned my addr 172.21.15.52:0/3206337736 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526180761 0 0) 0x7f565c0aecb0 con 0x7f565c0ac5a0 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5644003520 con 0x7f565c0ac5a0 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3804607688 0 0) 0x7f5644003520 con 0x7f565c0ac5a0 2024-03-29T11:20:36.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 >> v1:172.21.15.52:6790/0 conn(0x7f565c0b0770 legacy=0x7f565c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 >> v1:172.21.15.52:6789/0 conn(0x7f565c14a700 legacy=0x7f565c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 monclient: found mon.noname-c 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0ac9d0 con 0x7f565c0ac5a0 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f564c001670 con 0x7f565c0ac5a0 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: dump: 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:36.945 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f56613e2640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f56613e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f564c0018d0 con 0x7f565c0ac5a0 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 1 -- 172.21.15.52:0/3206337736 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f564c001670 con 0x7f565c0ac5a0 2024-03-29T11:20:36.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f5659ffb640 10 monclient: dump: 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f56613e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.964+0000 7f56613e2640 4 set_mon_vals no callback set 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.965+0000 7f56613e2640 10 monclient: shutdown 2024-03-29T11:20:36.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.965+0000 7f56613e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:36.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.965+0000 7f56613e2640 1 -- 172.21.15.52:0/3206337736 >> v1:172.21.15.79:6789/0 conn(0x7f565c0ac5a0 legacy=0x7f565c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.966+0000 7f56613e2640 1 -- 172.21.15.52:0/3206337736 shutdown_connections 2024-03-29T11:20:36.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.966+0000 7f56613e2640 1 -- 172.21.15.52:0/3206337736 >> 172.21.15.52:0/3206337736 conn(0x7f565c06b660 msgr2=0x7f565c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:36.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.967+0000 7f56613e2640 1 -- 172.21.15.52:0/3206337736 shutdown_connections 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.968+0000 7f56613e2640 1 -- 172.21.15.52:0/3206337736 wait complete. 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.968+0000 7f56613e2640 10 monclient: build_initial_monmap 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.969+0000 7f56613e2640 10 monclient: monmap: 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:36.950 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.969+0000 7f56613e2640 1 librados: starting msgr at 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.969+0000 7f56613e2640 1 librados: starting objecter 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.969+0000 7f56613e2640 1 Processor -- start 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 1 -- start start 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 1 librados: setting wanted keys 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 1 librados: calling monclient init 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient: init 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient(hunting): picked mon.noname-a con 0x7f565c0ac5a0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient(hunting): picked mon.noname-c con 0x7f565c0b0770 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient(hunting): picked mon.noname-b con 0x7f565c14a700 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c1239d0 con 0x7f565c0ac5a0 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c1263f0 con 0x7f565c14a700 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c145a30 con 0x7f565c0b0770 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.970+0000 7f56613e2640 10 monclient(hunting): authenticate will time out at 967.110992s 2024-03-29T11:20:36.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.975+0000 7f565b7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f565c14a700 0x7f565c145320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:59885/0 (socket says 172.21.15.52:59885) 2024-03-29T11:20:36.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.975+0000 7f565b7fe640 1 -- 172.21.15.52:0/4266380507 learned_addr learned my addr 172.21.15.52:0/4266380507 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:36.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.977+0000 7f56437fe640 10 client.?.objecter ms_handle_connect 0x7f565c14a700 2024-03-29T11:20:36.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.977+0000 7f56437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.977+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66724905 0 0) 0x7f565c1263f0 con 0x7f565c14a700 2024-03-29T11:20:36.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:20:36.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5648004eb0 con 0x7f565c14a700 2024-03-29T11:20:36.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 client.?.objecter ms_handle_connect 0x7f565c0ac5a0 2024-03-29T11:20:36.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1455824765 0 0) 0x7f5648004eb0 con 0x7f565c14a700 2024-03-29T11:20:36.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f565c1263f0 con 0x7f565c14a700 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f564c001670 con 0x7f565c14a700 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): dump: 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:36.958 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.978+0000 7f56437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.979+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1551393591 0 0) 0x7f565c1263f0 con 0x7f565c14a700 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.979+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 >> v1:172.21.15.52:6790/0 conn(0x7f565c0b0770 legacy=0x7f565c144c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 >> v1:172.21.15.52:6789/0 conn(0x7f565c0ac5a0 legacy=0x7f565c126880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 1 monclient: found mon.b 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c1284a0 con 0x7f565c14a700 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:20:36.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 10 monclient: _renew_subs 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c1dc220 con 0x7f565c14a700 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 10 monclient: _renew_subs 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c126020 con 0x7f565c14a700 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.980+0000 7f56613e2640 1 librados: init done 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.983+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f564c001c60 con 0x7f565c14a700 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.983+0000 7f56437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.983+0000 7f5658ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:36.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.983+0000 7f56613e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56613e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56613e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c005080 con 0x7f565c14a700 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f564c0050d0 con 0x7f565c14a700 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.984+0000 7f56437fe640 10 monclient: dump: 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:36.961 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:36.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.986+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f564c0052b0 con 0x7f565c14a700 2024-03-29T11:20:36.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f565affd640 10 monclient: get_auth_request con 0x7f564806de00 auth_method 0 2024-03-29T11:20:36.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f564c054850 con 0x7f565c14a700 2024-03-29T11:20:36.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f56437fe640 10 client.4240.objecter ms_dispatch 0x7f565c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:36.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f56437fe640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:36.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f56437fe640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:36.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f56437fe640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:20:36.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.987+0000 7f56437fe640 10 client.4240.objecter ms_handle_connect 0x7f564806de00 2024-03-29T11:20:36.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.992+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f564c050af0 con 0x7f565c14a700 2024-03-29T11:20:36.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.992+0000 7f56437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:36.992+0000 7f56437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:37.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.314+0000 7f56613e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:37.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.314+0000 7f56613e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:37.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.314+0000 7f56613e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:37.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.314+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f563c005330 con 0x7f565c14a700 2024-03-29T11:20:37.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.318+0000 7f56437fe640 1 -- 172.21.15.52:0/4266380507 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f564c050af0 con 0x7f565c14a700 2024-03-29T11:20:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.318+0000 7f56437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.318+0000 7f56437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:37.295 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:37.295 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:37.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 10 librados: watch_flush enter 2024-03-29T11:20:37.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 10 librados: watch_flush exit 2024-03-29T11:20:37.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:20:37.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:20:37.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 >> v1:172.21.15.79:6816/407480584 conn(0x7f564806de00 legacy=0x7f56480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:37.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 10 monclient: shutdown 2024-03-29T11:20:37.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:37.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.322+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 >> v1:172.21.15.79:6789/0 conn(0x7f565c14a700 legacy=0x7f565c145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:37.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.326+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 shutdown_connections 2024-03-29T11:20:37.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.326+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 >> 172.21.15.52:0/4266380507 conn(0x7f565c06b660 msgr2=0x7f565c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:37.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.327+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 shutdown_connections 2024-03-29T11:20:37.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.327+0000 7f56613e2640 1 -- 172.21.15.52:0/4266380507 wait complete. 2024-03-29T11:20:37.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:37.327+0000 7f56613e2640 1 librados: shutdown 2024-03-29T11:20:37.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:44.131 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:44.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.307+0000 7f98dfd6b640 10 monclient: get_monmap_and_config 2024-03-29T11:20:44.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.307+0000 7f98dfd6b640 10 monclient: build_initial_monmap 2024-03-29T11:20:44.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.307+0000 7f98dfd6b640 10 monclient: monmap: 2024-03-29T11:20:44.284 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:44.308046+0000 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:44.308046+0000 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:44.285 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.308+0000 7f98dfd6b640 1 Processor -- start 2024-03-29T11:20:44.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.308+0000 7f98dfd6b640 1 -- start start 2024-03-29T11:20:44.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.308+0000 7f98dfd6b640 10 monclient: init 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.309+0000 7f98dfd6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.309+0000 7f98dfd6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.309+0000 7f98dfd6b640 10 monclient(hunting): picked mon.noname-c con 0x7f98d80a9930 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f98d8149dc0 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 10 monclient(hunting): picked mon.noname-a con 0x7f98d814ff40 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d80aae60 con 0x7f98d814ff40 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d80b4310 con 0x7f98d80a9930 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d80afdd0 con 0x7f98d8149dc0 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:44.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.310+0000 7f98dfd6b640 10 monclient(hunting): authenticate will time out at 974.450370s 2024-03-29T11:20:44.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dd306640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f98d8149dc0 0x7f98d814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37169/0 (socket says 172.21.15.52:37169) 2024-03-29T11:20:44.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dd306640 1 -- 172.21.15.52:0/471751507 learned_addr learned my addr 172.21.15.52:0/471751507 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814912166 0 0) 0x7f98d80afdd0 con 0x7f98d8149dc0 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient(hunting): _negotiate 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4003520 con 0x7f98d8149dc0 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3134895385 0 0) 0x7f98c4003520 con 0x7f98d8149dc0 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 >> v1:172.21.15.79:6789/0 conn(0x7f98d80a9930 legacy=0x7f98d8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:44.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 >> v1:172.21.15.52:6789/0 conn(0x7f98d814ff40 legacy=0x7f98d8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient: _finish_hunting 0 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 monclient: found mon.noname-b 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d814a950 con 0x7f98d8149dc0 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient: _finish_auth 0 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 10 monclient: _check_auth_tickets 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f98d4001690 con 0x7f98d8149dc0 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dfd6b640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.313+0000 7f98dfd6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:44.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: dump: 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dfd6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d4001860 con 0x7f98d8149dc0 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 1 -- 172.21.15.52:0/471751507 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f98d4001cb0 con 0x7f98d8149dc0 2024-03-29T11:20:44.293 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dcb05640 10 monclient: dump: 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dfd6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:44.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.314+0000 7f98dfd6b640 4 set_mon_vals no callback set 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.316+0000 7f98dfd6b640 10 monclient: shutdown 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.316+0000 7f98dfd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.316+0000 7f98dfd6b640 1 -- 172.21.15.52:0/471751507 >> v1:172.21.15.52:6790/0 conn(0x7f98d8149dc0 legacy=0x7f98d814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.316+0000 7f98dfd6b640 1 -- 172.21.15.52:0/471751507 shutdown_connections 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.316+0000 7f98dfd6b640 1 -- 172.21.15.52:0/471751507 >> 172.21.15.52:0/471751507 conn(0x7f98d80b35e0 msgr2=0x7f98d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.316+0000 7f98dfd6b640 1 -- 172.21.15.52:0/471751507 shutdown_connections 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.317+0000 7f98dfd6b640 1 -- 172.21.15.52:0/471751507 wait complete. 2024-03-29T11:20:44.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 10 monclient: build_initial_monmap 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 10 monclient: monmap: 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 1 librados: starting msgr at 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 1 librados: starting objecter 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 1 Processor -- start 2024-03-29T11:20:44.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 1 -- start start 2024-03-29T11:20:44.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 1 librados: setting wanted keys 2024-03-29T11:20:44.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 1 librados: calling monclient init 2024-03-29T11:20:44.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.318+0000 7f98dfd6b640 10 monclient: init 2024-03-29T11:20:44.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.319+0000 7f98dfd6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:44.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.319+0000 7f98dfd6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:44.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.322+0000 7f98ddb07640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f98d80a9930 0x7f98d81dbe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:40859/0 (socket says 172.21.15.52:40859) 2024-03-29T11:20:44.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.323+0000 7f98ddb07640 1 -- 172.21.15.52:0/4099751460 learned_addr learned my addr 172.21.15.52:0/4099751460 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:44.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.323+0000 7f98dfd6b640 10 monclient(hunting): picked mon.noname-a con 0x7f98d80a9930 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:44.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.323+0000 7f98ce7fc640 10 client.?.objecter ms_handle_connect 0x7f98d80a9930 2024-03-29T11:20:44.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.323+0000 7f98ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f98d8149dc0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 10 monclient(hunting): picked mon.noname-c con 0x7f98d814ff40 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d80aca70 con 0x7f98d80a9930 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d81db560 con 0x7f98d8149dc0 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d81db6f0 con 0x7f98d814ff40 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:44.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.325+0000 7f98dfd6b640 10 monclient(hunting): authenticate will time out at 974.465851s 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 10 client.?.objecter ms_handle_connect 0x7f98d8149dc0 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176794207 0 0) 0x7f98d80aca70 con 0x7f98d80a9930 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:20:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.327+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98d80a9930 2024-03-29T11:20:44.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.329+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025044110 0 0) 0x7f98d81db560 con 0x7f98d8149dc0 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d80aca70 con 0x7f98d8149dc0 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 client.?.objecter ms_handle_connect 0x7f98d814ff40 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505971947 0 0) 0x7f98d81db6f0 con 0x7f98d814ff40 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:44.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.330+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d81db560 con 0x7f98d814ff40 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007052053 0 0) 0x7f98d80aca70 con 0x7f98d8149dc0 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d81db6f0 con 0x7f98d8149dc0 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f98d4001b70 con 0x7f98d8149dc0 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.311 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4011682723 0 0) 0x7f98bc004eb0 con 0x7f98d80a9930 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d80aca70 con 0x7f98d80a9930 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f98d0001ce0 con 0x7f98d80a9930 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.312 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3477474125 0 0) 0x7f98d81db6f0 con 0x7f98d8149dc0 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 >> v1:172.21.15.52:6790/0 conn(0x7f98d814ff40 legacy=0x7f98d8149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 >> v1:172.21.15.52:6789/0 conn(0x7f98d80a9930 legacy=0x7f98d81dbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.331+0000 7f98ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 1 monclient: found mon.b 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8149910 con 0x7f98d8149dc0 2024-03-29T11:20:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 10 monclient: _renew_subs 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d8149b60 con 0x7f98d8149dc0 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d4004bf0 con 0x7f98d8149dc0 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f98d4004f00 con 0x7f98d8149dc0 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98ce7fc640 10 monclient: dump: 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:44.314 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 10 monclient: _renew_subs 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d813f130 con 0x7f98d8149dc0 2024-03-29T11:20:44.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.332+0000 7f98dfd6b640 1 librados: init done 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.334+0000 7f98cf7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.335+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f98d401aed0 con 0x7f98d8149dc0 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.335+0000 7f98dfd6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.335+0000 7f98dfd6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.335+0000 7f98dfd6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.335+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c0005080 con 0x7f98d8149dc0 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.336+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f98d4084a40 con 0x7f98d8149dc0 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.336+0000 7f98ce7fc640 10 client.4243.objecter ms_dispatch 0x7f98d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.336+0000 7f98ce7fc640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.336+0000 7f98ce7fc640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.336+0000 7f98ce7fc640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:20:44.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.337+0000 7f98ddb07640 10 monclient: get_auth_request con 0x7f98bc06dd30 auth_method 0 2024-03-29T11:20:44.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.340+0000 7f98ce7fc640 10 client.4243.objecter ms_handle_connect 0x7f98bc06dd30 2024-03-29T11:20:44.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.345+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98d4050320 con 0x7f98d8149dc0 2024-03-29T11:20:44.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.345+0000 7f98ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:44.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.345+0000 7f98ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:44.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.673+0000 7f98dfd6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.673+0000 7f98dfd6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.673+0000 7f98dfd6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:20:44.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.673+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98c0005330 con 0x7f98d8149dc0 2024-03-29T11:20:44.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.675+0000 7f98ce7fc640 1 -- 172.21.15.52:0/4099751460 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f98d4052250 con 0x7f98d8149dc0 2024-03-29T11:20:44.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.675+0000 7f98ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.675+0000 7f98ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:44.653 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:44.653 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:44.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.678+0000 7f98dfd6b640 10 librados: watch_flush enter 2024-03-29T11:20:44.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.679+0000 7f98dfd6b640 10 librados: watch_flush exit 2024-03-29T11:20:44.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.679+0000 7f98dfd6b640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:20:44.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.679+0000 7f98dfd6b640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:20:44.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.680+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 >> v1:172.21.15.79:6816/407480584 conn(0x7f98bc06dd30 legacy=0x7f98bc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.680+0000 7f98dfd6b640 10 monclient: shutdown 2024-03-29T11:20:44.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.680+0000 7f98dfd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:44.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.680+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 >> v1:172.21.15.79:6789/0 conn(0x7f98d8149dc0 legacy=0x7f98d81487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.680+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 shutdown_connections 2024-03-29T11:20:44.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.680+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 >> 172.21.15.52:0/4099751460 conn(0x7f98d80b35e0 msgr2=0x7f98d80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:44.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.681+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 shutdown_connections 2024-03-29T11:20:44.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.681+0000 7f98dfd6b640 1 -- 172.21.15.52:0/4099751460 wait complete. 2024-03-29T11:20:44.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:44.681+0000 7f98dfd6b640 1 librados: shutdown 2024-03-29T11:20:44.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:51.481 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:51.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.662+0000 7f39bdc63640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.663+0000 7f39bdc63640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.663+0000 7f39bdc63640 10 monclient: monmap: 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:51.663940+0000 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:51.663940+0000 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:51.640 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.665+0000 7f39bdc63640 1 Processor -- start 2024-03-29T11:20:51.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.665+0000 7f39bdc63640 1 -- start start 2024-03-29T11:20:51.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.665+0000 7f39bdc63640 10 monclient: init 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient(hunting): picked mon.noname-a con 0x7f39b81422b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient(hunting): picked mon.noname-b con 0x7f39b8142690 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient(hunting): picked mon.noname-c con 0x7f39b80b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b8142c80 con 0x7f39b81422b0 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80aae40 con 0x7f39b80b17d0 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b813fbd0 con 0x7f39b8142690 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.666+0000 7f39bdc63640 10 monclient(hunting): authenticate will time out at 981.806556s 2024-03-29T11:20:51.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.669+0000 7f39b77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f39b81422b0 0x7f39b813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:43819/0 (socket says 172.21.15.52:43819) 2024-03-29T11:20:51.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.669+0000 7f39b77fe640 1 -- 172.21.15.52:0/2635199858 learned_addr learned my addr 172.21.15.52:0/2635199858 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:51.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.674+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466237637 0 0) 0x7f39b8142c80 con 0x7f39b81422b0 2024-03-29T11:20:51.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.674+0000 7f39b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.674+0000 7f39b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.674+0000 7f39b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.674+0000 7f39b67fc640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.674+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a0003520 con 0x7f39b81422b0 2024-03-29T11:20:51.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.677+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775817449 0 0) 0x7f39b813fbd0 con 0x7f39b8142690 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.677+0000 7f39b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b8142c80 con 0x7f39b8142690 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752529852 0 0) 0x7f39b80aae40 con 0x7f39b80b17d0 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b813fbd0 con 0x7f39b80b17d0 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244813684 0 0) 0x7f39a0003520 con 0x7f39b81422b0 2024-03-29T11:20:51.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 >> v1:172.21.15.52:6790/0 conn(0x7f39b8142690 legacy=0x7f39b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 >> v1:172.21.15.79:6789/0 conn(0x7f39b80b17d0 legacy=0x7f39b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.678+0000 7f39b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.679+0000 7f39b67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.679+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b8151940 con 0x7f39b81422b0 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.679+0000 7f39b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.679+0000 7f39b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.679+0000 7f39b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39bdc63640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39bdc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f39ac001610 con 0x7f39b81422b0 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39b67fc640 10 monclient: dump: 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.680+0000 7f39bdc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.681+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39ac001830 con 0x7f39b81422b0 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.681+0000 7f39b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.681+0000 7f39b67fc640 1 -- 172.21.15.52:0/2635199858 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f39ac001bc0 con 0x7f39b81422b0 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.681+0000 7f39b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.682+0000 7f39b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.682+0000 7f39b67fc640 10 monclient: dump: 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:51.660 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.682+0000 7f39bdc63640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.682+0000 7f39bdc63640 4 set_mon_vals no callback set 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.683+0000 7f39bdc63640 10 monclient: shutdown 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.683+0000 7f39bdc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.683+0000 7f39bdc63640 1 -- 172.21.15.52:0/2635199858 >> v1:172.21.15.52:6789/0 conn(0x7f39b81422b0 legacy=0x7f39b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.686+0000 7f39bdc63640 1 -- 172.21.15.52:0/2635199858 shutdown_connections 2024-03-29T11:20:51.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.686+0000 7f39bdc63640 1 -- 172.21.15.52:0/2635199858 >> 172.21.15.52:0/2635199858 conn(0x7f39b8066a00 msgr2=0x7f39b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.688+0000 7f39bdc63640 1 -- 172.21.15.52:0/2635199858 shutdown_connections 2024-03-29T11:20:51.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.688+0000 7f39bdc63640 1 -- 172.21.15.52:0/2635199858 wait complete. 2024-03-29T11:20:51.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.689+0000 7f39bdc63640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.689+0000 7f39bdc63640 10 monclient: monmap: 2024-03-29T11:20:51.666 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:51.666 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.689+0000 7f39bdc63640 1 librados: starting msgr at 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.689+0000 7f39bdc63640 1 librados: starting objecter 2024-03-29T11:20:51.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.689+0000 7f39bdc63640 1 Processor -- start 2024-03-29T11:20:51.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.690+0000 7f39bdc63640 1 -- start start 2024-03-29T11:20:51.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.690+0000 7f39bdc63640 1 librados: setting wanted keys 2024-03-29T11:20:51.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.690+0000 7f39bdc63640 1 librados: calling monclient init 2024-03-29T11:20:51.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.690+0000 7f39bdc63640 10 monclient: init 2024-03-29T11:20:51.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient(hunting): picked mon.noname-c con 0x7f39b80b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient(hunting): picked mon.noname-b con 0x7f39b81422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient(hunting): picked mon.noname-a con 0x7f39b8142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b806e6f0 con 0x7f39b8142690 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80b6900 con 0x7f39b81422b0 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b812e870 con 0x7f39b80b17d0 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.691+0000 7f39bdc63640 10 monclient(hunting): authenticate will time out at 981.831979s 2024-03-29T11:20:51.672 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.695+0000 7f39b7fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f39b8142690 0x7f39b812e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:43981/0 (socket says 172.21.15.52:43981) 2024-03-29T11:20:51.672 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.695+0000 7f39b7fff640 1 -- 172.21.15.52:0/373493144 learned_addr learned my addr 172.21.15.52:0/373493144 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.695+0000 7f399ffff640 10 client.?.objecter ms_handle_connect 0x7f39b8142690 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.695+0000 7f399ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208126088 0 0) 0x7f39b806e6f0 con 0x7f39b8142690 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:20:51.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a4004eb0 con 0x7f39b8142690 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.696+0000 7f399ffff640 10 client.?.objecter ms_handle_connect 0x7f39b80b17d0 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625322758 0 0) 0x7f39b812e870 con 0x7f39b80b17d0 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:20:51.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.697+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b806e6f0 con 0x7f39b80b17d0 2024-03-29T11:20:51.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.698+0000 7f399ffff640 10 client.?.objecter ms_handle_connect 0x7f39b81422b0 2024-03-29T11:20:51.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.698+0000 7f399ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.698+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 262004364 0 0) 0x7f39b806e6f0 con 0x7f39b80b17d0 2024-03-29T11:20:51.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.698+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39b812e870 con 0x7f39b80b17d0 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.700+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f39ac001550 con 0x7f39b80b17d0 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.700+0000 7f399ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.700+0000 7f399ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.700+0000 7f399ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.682 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104531687 0 0) 0x7f39b80b6900 con 0x7f39b81422b0 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b806e6f0 con 0x7f39b81422b0 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3432832265 0 0) 0x7f39a4004eb0 con 0x7f39b8142690 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39b80b6900 con 0x7f39b8142690 2024-03-29T11:20:51.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f39b0001c90 con 0x7f39b8142690 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.684 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2341033010 0 0) 0x7f39b80b6900 con 0x7f39b8142690 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.701+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 >> v1:172.21.15.52:6790/0 conn(0x7f39b80b17d0 legacy=0x7f39b80b6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 >> v1:172.21.15.79:6789/0 conn(0x7f39b81422b0 legacy=0x7f39b812da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 1 monclient: found mon.a 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b806e940 con 0x7f39b8142690 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 10 monclient: _renew_subs 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b806ec20 con 0x7f39b8142690 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 10 monclient: _renew_subs 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b80b74a0 con 0x7f39b8142690 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39bdc63640 1 librados: init done 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39b0003990 con 0x7f39b8142690 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f39b5ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.702+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f39b0003de0 con 0x7f39b8142690 2024-03-29T11:20:51.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f399ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f399ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f399ffff640 10 monclient: dump: 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f39bdc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f39bdc63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f39bdc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.703+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998005080 con 0x7f39b8142690 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.704+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f39b0003f40 con 0x7f39b8142690 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f39b77fe640 10 monclient: get_auth_request con 0x7f39a406de30 auth_method 0 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f39b0084da0 con 0x7f39b8142690 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f399ffff640 10 client.4275.objecter ms_dispatch 0x7f39b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f399ffff640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f399ffff640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f399ffff640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.705+0000 7f399ffff640 10 client.4275.objecter ms_handle_connect 0x7f39a406de30 2024-03-29T11:20:51.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.717+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39b0050680 con 0x7f39b8142690 2024-03-29T11:20:51.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.717+0000 7f399ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:51.717+0000 7f399ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:52.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.043+0000 7f39bdc63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:52.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.043+0000 7f39bdc63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:52.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.043+0000 7f39bdc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:20:52.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.043+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3998005330 con 0x7f39b8142690 2024-03-29T11:20:52.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.045+0000 7f399ffff640 1 -- 172.21.15.52:0/373493144 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f39b00525b0 con 0x7f39b8142690 2024-03-29T11:20:52.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.045+0000 7f399ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:52.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.045+0000 7f399ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:52.023 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:52.023 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 10 librados: watch_flush enter 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 10 librados: watch_flush exit 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 10 client.4275.objecter successfully canceled tick 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 >> v1:172.21.15.79:6816/407480584 conn(0x7f39a406de30 legacy=0x7f39a40702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 10 monclient: shutdown 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.049+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 >> v1:172.21.15.52:6789/0 conn(0x7f39b8142690 legacy=0x7f39b812e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.050+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 shutdown_connections 2024-03-29T11:20:52.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.050+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 >> 172.21.15.52:0/373493144 conn(0x7f39b8066a00 msgr2=0x7f39b80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:52.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.052+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 shutdown_connections 2024-03-29T11:20:52.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.052+0000 7f39bdc63640 1 -- 172.21.15.52:0/373493144 wait complete. 2024-03-29T11:20:52.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:52.052+0000 7f39bdc63640 1 librados: shutdown 2024-03-29T11:20:52.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:58.853 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:59.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.030+0000 7f210b1cb640 10 monclient: get_monmap_and_config 2024-03-29T11:20:59.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.030+0000 7f210b1cb640 10 monclient: build_initial_monmap 2024-03-29T11:20:59.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.030+0000 7f210b1cb640 10 monclient: monmap: 2024-03-29T11:20:59.007 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:59.007 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:59.007 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:20:59.031605+0000 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:20:59.031605+0000 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:20:59.008 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:59.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.032+0000 7f210b1cb640 1 Processor -- start 2024-03-29T11:20:59.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.032+0000 7f210b1cb640 1 -- start start 2024-03-29T11:20:59.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.032+0000 7f210b1cb640 10 monclient: init 2024-03-29T11:20:59.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.033+0000 7f210b1cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.033+0000 7f210b1cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.033+0000 7f210b1cb640 10 monclient(hunting): picked mon.noname-b con 0x7f21040b1f60 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.033+0000 7f210b1cb640 10 monclient(hunting): picked mon.noname-c con 0x7f2104071eb0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.035+0000 7f210b1cb640 10 monclient(hunting): picked mon.noname-a con 0x7f2104143d10 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.035+0000 7f210b1cb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210406ef80 con 0x7f2104143d10 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.035+0000 7f210b1cb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210406f260 con 0x7f2104071eb0 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.035+0000 7f210b1cb640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21040b40c0 con 0x7f21040b1f60 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.035+0000 7f210b1cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:59.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.035+0000 7f210b1cb640 10 monclient(hunting): authenticate will time out at 989.175216s 2024-03-29T11:20:59.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.038+0000 7f2109768640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f2104143d10 0x7f2104146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:44537/0 (socket says 172.21.15.52:44537) 2024-03-29T11:20:59.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.038+0000 7f2109768640 1 -- 172.21.15.52:0/3746633980 learned_addr learned my addr 172.21.15.52:0/3746633980 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:59.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556910557 0 0) 0x7f210406ef80 con 0x7f2104143d10 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4003520 con 0x7f2104143d10 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702164803 0 0) 0x7f210406f260 con 0x7f2104071eb0 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210406ef80 con 0x7f2104071eb0 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236344809 0 0) 0x7f20f4003520 con 0x7f2104143d10 2024-03-29T11:20:59.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 >> v1:172.21.15.52:6790/0 conn(0x7f21040b1f60 legacy=0x7f2104143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 >> v1:172.21.15.79:6789/0 conn(0x7f2104071eb0 legacy=0x7f2104146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 monclient: found mon.noname-a 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104155d70 con 0x7f2104143d10 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f210b1cb640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f210b1cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f20fc001a10 con 0x7f2104143d10 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: dump: 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:59.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f210b1cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 1 -- 172.21.15.52:0/3746633980 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20fc001c30 con 0x7f2104143d10 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f21037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f210b1cb640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.039+0000 7f210b1cb640 4 set_mon_vals no callback set 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.040+0000 7f210b1cb640 10 monclient: shutdown 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.040+0000 7f210b1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.040+0000 7f210b1cb640 1 -- 172.21.15.52:0/3746633980 >> v1:172.21.15.52:6789/0 conn(0x7f2104143d10 legacy=0x7f2104146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.040+0000 7f210b1cb640 1 -- 172.21.15.52:0/3746633980 shutdown_connections 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.040+0000 7f210b1cb640 1 -- 172.21.15.52:0/3746633980 >> 172.21.15.52:0/3746633980 conn(0x7f2104066a00 msgr2=0x7f2104066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.040+0000 7f210b1cb640 1 -- 172.21.15.52:0/3746633980 shutdown_connections 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.041+0000 7f210b1cb640 1 -- 172.21.15.52:0/3746633980 wait complete. 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.043+0000 7f210b1cb640 10 monclient: build_initial_monmap 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.043+0000 7f210b1cb640 10 monclient: monmap: 2024-03-29T11:20:59.023 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:20:59.024 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 1 librados: starting msgr at 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 1 librados: starting objecter 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 1 Processor -- start 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 1 -- start start 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 1 librados: setting wanted keys 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 1 librados: calling monclient init 2024-03-29T11:20:59.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.045+0000 7f210b1cb640 10 monclient: init 2024-03-29T11:20:59.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.046+0000 7f210b1cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:59.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.046+0000 7f210b1cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:59.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.046+0000 7f210b1cb640 10 monclient(hunting): picked mon.noname-a con 0x7f21040b1f60 addr v1:172.21.15.52:6789/0 2024-03-29T11:20:59.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.046+0000 7f210b1cb640 10 monclient(hunting): picked mon.noname-b con 0x7f21041421f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:20:59.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.050+0000 7f210b1cb640 10 monclient(hunting): picked mon.noname-c con 0x7f21041536e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:20:59.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.050+0000 7f210b1cb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104142fc0 con 0x7f21040b1f60 2024-03-29T11:20:59.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.050+0000 7f210b1cb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104142c90 con 0x7f21041421f0 2024-03-29T11:20:59.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.050+0000 7f210b1cb640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104154840 con 0x7f21041536e0 2024-03-29T11:20:59.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.050+0000 7f210b1cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:59.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.050+0000 7f210b1cb640 10 monclient(hunting): authenticate will time out at 989.190474s 2024-03-29T11:20:59.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.051+0000 7f2109768640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f21041536e0 0x7f21041deec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52221/0 (socket says 172.21.15.52:52221) 2024-03-29T11:20:59.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.051+0000 7f2109768640 1 -- 172.21.15.52:0/2776913543 learned_addr learned my addr 172.21.15.52:0/2776913543 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:20:59.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 client.?.objecter ms_handle_connect 0x7f21041536e0 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217224465 0 0) 0x7f2104154840 con 0x7f21041536e0 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8004eb0 con 0x7f21041536e0 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3833531105 0 0) 0x7f20e8004eb0 con 0x7f21041536e0 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2104154840 con 0x7f21041536e0 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f20fc0013a0 con 0x7f21041536e0 2024-03-29T11:20:59.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): dump: 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:59.034 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 client.?.objecter ms_handle_connect 0x7f21040b1f60 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.052+0000 7f21017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135587815 0 0) 0x7f2104142fc0 con 0x7f21040b1f60 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:59.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8004eb0 con 0x7f21040b1f60 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1794165953 0 0) 0x7f2104154840 con 0x7f21041536e0 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.053+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 >> v1:172.21.15.79:6789/0 conn(0x7f21041421f0 legacy=0x7f2104145ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.054+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 >> v1:172.21.15.52:6789/0 conn(0x7f21040b1f60 legacy=0x7f2104154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 1 monclient: found mon.c 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104154b20 con 0x7f21041536e0 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 10 monclient: _renew_subs 2024-03-29T11:20:59.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210406f5d0 con 0x7f21041536e0 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 10 monclient: _renew_subs 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21041df7b0 con 0x7f21041536e0 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f210b1cb640 1 librados: init done 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20fc001e20 con 0x7f21041536e0 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f2102ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:59.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f20fc005150 con 0x7f21041536e0 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.055+0000 7f21017fa640 10 monclient: dump: 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:20:59.038 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:20:59.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.057+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f20fc01b120 con 0x7f21041536e0 2024-03-29T11:20:59.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.057+0000 7f210b1cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:59.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.057+0000 7f210b1cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.057+0000 7f210b1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.057+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e4005080 con 0x7f21041536e0 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.058+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f20fc084ef0 con 0x7f21041536e0 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.058+0000 7f21017fa640 10 client.4277.objecter ms_dispatch 0x7f2104000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.058+0000 7f21017fa640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.058+0000 7f21017fa640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.058+0000 7f21017fa640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T11:20:59.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.061+0000 7f2108f67640 10 monclient: get_auth_request con 0x7f20e806db10 auth_method 0 2024-03-29T11:20:59.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.063+0000 7f21017fa640 10 client.4277.objecter ms_handle_connect 0x7f20e806db10 2024-03-29T11:20:59.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.066+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20fc0507d0 con 0x7f21041536e0 2024-03-29T11:20:59.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.066+0000 7f21017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:59.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.066+0000 7f21017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:59.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.399+0000 7f210b1cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:59.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.399+0000 7f210b1cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:59.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.399+0000 7f210b1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:20:59.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.399+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20e4005330 con 0x7f21041536e0 2024-03-29T11:20:59.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.400+0000 7f21017fa640 1 -- 172.21.15.52:0/2776913543 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f20fc052700 con 0x7f21041536e0 2024-03-29T11:20:59.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.401+0000 7f21017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:59.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.401+0000 7f21017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:59.378 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:20:59.378 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:59.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 10 librados: watch_flush enter 2024-03-29T11:20:59.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 10 librados: watch_flush exit 2024-03-29T11:20:59.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T11:20:59.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 10 client.4277.objecter successfully canceled tick 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 >> v1:172.21.15.79:6816/407480584 conn(0x7f20e806db10 legacy=0x7f20e806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 10 monclient: shutdown 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.405+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 >> v1:172.21.15.52:6790/0 conn(0x7f21041536e0 legacy=0x7f21041deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.406+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 shutdown_connections 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.406+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 >> 172.21.15.52:0/2776913543 conn(0x7f2104066a00 msgr2=0x7f21041467c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.406+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 shutdown_connections 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.406+0000 7f210b1cb640 1 -- 172.21.15.52:0/2776913543 wait complete. 2024-03-29T11:20:59.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:20:59.406+0000 7f210b1cb640 1 librados: shutdown 2024-03-29T11:20:59.401 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:06.208 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:06.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.382+0000 7f30b50f6640 10 monclient: get_monmap_and_config 2024-03-29T11:21:06.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.382+0000 7f30b50f6640 10 monclient: build_initial_monmap 2024-03-29T11:21:06.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.383+0000 7f30b50f6640 10 monclient: monmap: 2024-03-29T11:21:06.359 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:06.359 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:06.383935+0000 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:06.383935+0000 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:06.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.384+0000 7f30b50f6640 1 Processor -- start 2024-03-29T11:21:06.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.384+0000 7f30b50f6640 1 -- start start 2024-03-29T11:21:06.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.384+0000 7f30b50f6640 10 monclient: init 2024-03-29T11:21:06.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:06.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:06.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient(hunting): picked mon.noname-b con 0x7f30b00b4710 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient(hunting): picked mon.noname-c con 0x7f30b00b3970 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient(hunting): picked mon.noname-a con 0x7f30b013f390 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00af020 con 0x7f30b013f390 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00b0910 con 0x7f30b00b3970 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00b0bf0 con 0x7f30b00b4710 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:06.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.385+0000 7f30b50f6640 10 monclient(hunting): authenticate will time out at 996.526036s 2024-03-29T11:21:06.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.391+0000 7f30ae59c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f30b00b3970 0x7f30b00af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:46531/0 (socket says 172.21.15.52:46531) 2024-03-29T11:21:06.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.391+0000 7f30ae59c640 1 -- 172.21.15.52:0/3857492025 learned_addr learned my addr 172.21.15.52:0/3857492025 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:06.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.393+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771868679 0 0) 0x7f30b00b0910 con 0x7f30b00b3970 2024-03-29T11:21:06.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.393+0000 7f30add9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:06.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.393+0000 7f30add9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:06.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.393+0000 7f30add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:06.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.393+0000 7f30add9b640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:21:06.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.393+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003520 con 0x7f30b00b3970 2024-03-29T11:21:06.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2255235199 0 0) 0x7f3098003520 con 0x7f30b00b3970 2024-03-29T11:21:06.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 >> v1:172.21.15.52:6790/0 conn(0x7f30b00b4710 legacy=0x7f30b00b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 >> v1:172.21.15.52:6789/0 conn(0x7f30b013f390 legacy=0x7f30b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 monclient: found mon.noname-c 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b00aeb00 con 0x7f30b00b3970 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30a8001a10 con 0x7f30b00b3970 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:06.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: dump: 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a8001ca0 con 0x7f30b00b3970 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 1 -- 172.21.15.52:0/3857492025 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30a8002030 con 0x7f30b00b3970 2024-03-29T11:21:06.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.394+0000 7f30add9b640 10 monclient: dump: 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:06.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.396+0000 7f30b50f6640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:21:06.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.396+0000 7f30b50f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:06.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.396+0000 7f30b50f6640 4 set_mon_vals no callback set 2024-03-29T11:21:06.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.398+0000 7f30b50f6640 10 monclient: shutdown 2024-03-29T11:21:06.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.398+0000 7f30b50f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:06.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.398+0000 7f30b50f6640 1 -- 172.21.15.52:0/3857492025 >> v1:172.21.15.79:6789/0 conn(0x7f30b00b3970 legacy=0x7f30b00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.398+0000 7f30b50f6640 1 -- 172.21.15.52:0/3857492025 shutdown_connections 2024-03-29T11:21:06.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.398+0000 7f30b50f6640 1 -- 172.21.15.52:0/3857492025 >> 172.21.15.52:0/3857492025 conn(0x7f30b0066a00 msgr2=0x7f30b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:06.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.399+0000 7f30b50f6640 1 -- 172.21.15.52:0/3857492025 shutdown_connections 2024-03-29T11:21:06.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.400+0000 7f30b50f6640 1 -- 172.21.15.52:0/3857492025 wait complete. 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.401+0000 7f30b50f6640 10 monclient: build_initial_monmap 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.401+0000 7f30b50f6640 10 monclient: monmap: 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:06.378 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:06.379 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:06.379 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:06.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.401+0000 7f30b50f6640 1 librados: starting msgr at 2024-03-29T11:21:06.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.401+0000 7f30b50f6640 1 librados: starting objecter 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.402+0000 7f30b50f6640 1 Processor -- start 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.402+0000 7f30b50f6640 1 -- start start 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.403+0000 7f30b50f6640 1 librados: setting wanted keys 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.403+0000 7f30b50f6640 1 librados: calling monclient init 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.403+0000 7f30b50f6640 10 monclient: init 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:06.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient(hunting): picked mon.noname-b con 0x7f30b00b4710 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient(hunting): picked mon.noname-a con 0x7f30b013f390 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient(hunting): picked mon.noname-c con 0x7f30b01354c0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b013d460 con 0x7f30b013f390 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b0134660 con 0x7f30b00b4710 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b013b5b0 con 0x7f30b01354c0 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:06.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.404+0000 7f30b50f6640 10 monclient(hunting): authenticate will time out at 996.545037s 2024-03-29T11:21:06.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.408+0000 7f30ae59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f30b013f390 0x7f30b0133840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:40411/0 (socket says 172.21.15.52:40411) 2024-03-29T11:21:06.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.408+0000 7f30ae59c640 1 -- 172.21.15.52:0/2894018346 learned_addr learned my addr 172.21.15.52:0/2894018346 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:06.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.408+0000 7f30977fe640 10 client.?.objecter ms_handle_connect 0x7f30b00b4710 2024-03-29T11:21:06.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.408+0000 7f30977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:06.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 client.?.objecter ms_handle_connect 0x7f30b013f390 2024-03-29T11:21:06.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:06.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216364316 0 0) 0x7f30b0134660 con 0x7f30b00b4710 2024-03-29T11:21:06.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:06.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f309c004eb0 con 0x7f30b00b4710 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319625801 0 0) 0x7f30b013d460 con 0x7f30b013f390 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.409+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0134660 con 0x7f30b013f390 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1515208985 0 0) 0x7f30b0134660 con 0x7f30b013f390 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f309c005490 con 0x7f30b013f390 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30a80018d0 con 0x7f30b013f390 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:06.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 10 monclient(hunting): dump: 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:06.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.410+0000 7f30977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 10 client.?.objecter ms_handle_connect 0x7f30b01354c0 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489604576 0 0) 0x7f309c005490 con 0x7f30b013f390 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 >> v1:172.21.15.52:6790/0 conn(0x7f30b01354c0 legacy=0x7f30b0133f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 >> v1:172.21.15.79:6789/0 conn(0x7f30b00b4710 legacy=0x7f30b01399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:06.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 1 monclient: found mon.a 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b013b890 con 0x7f30b013f390 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.412+0000 7f30977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 10 monclient: _renew_subs 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b01348e0 con 0x7f30b013f390 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 10 monclient: _renew_subs 2024-03-29T11:21:06.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b013b130 con 0x7f30b013f390 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.413+0000 7f30b50f6640 1 librados: init done 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.415+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a8001ab0 con 0x7f30b013f390 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.415+0000 7f30977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.416+0000 7f30ad59a640 4 set_mon_vals no callback set 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.416+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30a80018d0 con 0x7f30b013f390 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.416+0000 7f30977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.416+0000 7f30977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.416+0000 7f30977fe640 10 monclient: dump: 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:06.395 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:06.396 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:06.396 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:06.396 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:06.396 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:06.396 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.417+0000 7f30b50f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.417+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f30a801ade0 con 0x7f30b013f390 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.418+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f30a80546f0 con 0x7f30b013f390 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.418+0000 7f30977fe640 10 client.4287.objecter ms_dispatch 0x7f30b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.418+0000 7f30977fe640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.418+0000 7f30977fe640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:06.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.418+0000 7f30977fe640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-29T11:21:06.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.417+0000 7f30b50f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:06.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.419+0000 7f30b50f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:06.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.419+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3090005080 con 0x7f30b013f390 2024-03-29T11:21:06.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.421+0000 7f30aed9d640 10 monclient: get_auth_request con 0x7f309c06dfb0 auth_method 0 2024-03-29T11:21:06.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.422+0000 7f30977fe640 10 client.4287.objecter ms_handle_connect 0x7f309c06dfb0 2024-03-29T11:21:06.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.427+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30a8050990 con 0x7f30b013f390 2024-03-29T11:21:06.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.427+0000 7f30977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:06.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.427+0000 7f30977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:06.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.758+0000 7f30b50f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.758+0000 7f30b50f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.758+0000 7f30b50f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:06.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.758+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3090005330 con 0x7f30b013f390 2024-03-29T11:21:06.737 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:06.737 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:06.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.759+0000 7f30977fe640 1 -- 172.21.15.52:0/2894018346 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f30a8050990 con 0x7f30b013f390 2024-03-29T11:21:06.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.759+0000 7f30977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.759+0000 7f30977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 10 librados: watch_flush enter 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 10 librados: watch_flush exit 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 10 client.4287.objecter successfully canceled tick 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 >> v1:172.21.15.79:6816/407480584 conn(0x7f309c06dfb0 legacy=0x7f309c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 10 monclient: shutdown 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 >> v1:172.21.15.52:6789/0 conn(0x7f30b013f390 legacy=0x7f30b0133840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 shutdown_connections 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 >> 172.21.15.52:0/2894018346 conn(0x7f30b0066a00 msgr2=0x7f30b013c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 shutdown_connections 2024-03-29T11:21:06.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 -- 172.21.15.52:0/2894018346 wait complete. 2024-03-29T11:21:06.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:06.763+0000 7f30b50f6640 1 librados: shutdown 2024-03-29T11:21:06.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:13.565 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:13.721 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.744+0000 7f4161e0d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:13.721 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.744+0000 7f4161e0d640 10 monclient: build_initial_monmap 2024-03-29T11:21:13.721 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.744+0000 7f4161e0d640 10 monclient: monmap: 2024-03-29T11:21:13.721 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:13.721 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:13.745828+0000 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:13.745828+0000 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:13.722 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.746+0000 7f4161e0d640 1 Processor -- start 2024-03-29T11:21:13.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.746+0000 7f4161e0d640 1 -- start start 2024-03-29T11:21:13.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.746+0000 7f4161e0d640 10 monclient: init 2024-03-29T11:21:13.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.747+0000 7f4161e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:13.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.747+0000 7f4161e0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:13.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.747+0000 7f4161e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f415c1422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f415c142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f415b7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f415c1422b0 0x7f415c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:36570/0 (socket says 172.21.15.52:36570) 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f415b7fe640 1 -- 172.21.15.52:0/850803090 learned_addr learned my addr 172.21.15.52:0/850803090 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f415c0b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c142c80 con 0x7f415c142690 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0aae40 con 0x7f415c0b17d0 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c13fbd0 con 0x7f415c1422b0 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:13.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.748+0000 7f4161e0d640 10 monclient(hunting): authenticate will time out at 1003.888780s 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302602809 0 0) 0x7f415c0aae40 con 0x7f415c0b17d0 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144003520 con 0x7f415c0b17d0 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003658646 0 0) 0x7f415c13fbd0 con 0x7f415c1422b0 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0aae40 con 0x7f415c1422b0 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275223087 0 0) 0x7f4144003520 con 0x7f415c0b17d0 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 >> v1:172.21.15.52:6790/0 conn(0x7f415c1422b0 legacy=0x7f415c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 >> v1:172.21.15.52:6789/0 conn(0x7f415c142690 legacy=0x7f415c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c151940 con 0x7f415c0b17d0 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:13.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4154001a10 con 0x7f415c0b17d0 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f415a7fc640 10 monclient: dump: 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:13.730 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f4161e0d640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.749+0000 7f4161e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4154001b70 con 0x7f415c0b17d0 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f415a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f415a7fc640 1 -- 172.21.15.52:0/850803090 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4154001f00 con 0x7f415c0b17d0 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f415a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f415a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f415a7fc640 10 monclient: dump: 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.731 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f4161e0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.750+0000 7f4161e0d640 4 set_mon_vals no callback set 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.751+0000 7f4161e0d640 10 monclient: shutdown 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.751+0000 7f4161e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.751+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 >> v1:172.21.15.79:6789/0 conn(0x7f415c0b17d0 legacy=0x7f415c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.752+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 shutdown_connections 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.752+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 >> 172.21.15.52:0/850803090 conn(0x7f415c066a00 msgr2=0x7f415c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.752+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 shutdown_connections 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.752+0000 7f4161e0d640 1 -- 172.21.15.52:0/850803090 wait complete. 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.753+0000 7f4161e0d640 10 monclient: build_initial_monmap 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.753+0000 7f4161e0d640 10 monclient: monmap: 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:13.741 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 1 librados: starting msgr at 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 1 librados: starting objecter 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 1 Processor -- start 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 1 -- start start 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 1 librados: setting wanted keys 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 1 librados: calling monclient init 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.754+0000 7f4161e0d640 10 monclient: init 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.755+0000 7f4161e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.755+0000 7f4161e0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.755+0000 7f4161e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f415c0b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f415c1422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f415b7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f415c0b17d0 0x7f415c0aee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:44916/0 (socket says 172.21.15.52:44916) 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f415c142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:13.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f415affd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f415c1422b0 0x7f415c13abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:36790/0 (socket says 172.21.15.52:36790) 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0b0770 con 0x7f415c142690 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f415affd640 1 -- 172.21.15.52:0/3076848452 learned_addr learned my addr 172.21.15.52:0/3076848452 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c13b9d0 con 0x7f415c0b17d0 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0abae0 con 0x7f415c1422b0 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4161e0d640 10 monclient(hunting): authenticate will time out at 1003.896366s 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 client.?.objecter ms_handle_connect 0x7f415c0b17d0 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 client.?.objecter ms_handle_connect 0x7f415c142690 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 client.?.objecter ms_handle_connect 0x7f415c1422b0 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377086823 0 0) 0x7f415c13b9d0 con 0x7f415c0b17d0 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4148004eb0 con 0x7f415c0b17d0 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842566680 0 0) 0x7f415c0abae0 con 0x7f415c1422b0 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c13b9d0 con 0x7f415c1422b0 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415885562 0 0) 0x7f415c0b0770 con 0x7f415c142690 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.756+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0abae0 con 0x7f415c142690 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785782288 0 0) 0x7f415c0abae0 con 0x7f415c142690 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c0b0770 con 0x7f415c142690 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4154001480 con 0x7f415c142690 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): dump: 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:13.745 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2534627495 0 0) 0x7f4148004eb0 con 0x7f415c0b17d0 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c0abae0 con 0x7f415c0b17d0 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f414c001930 con 0x7f415c0b17d0 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.757+0000 7f4143fff640 10 monclient(hunting): dump: 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.746 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3290816464 0 0) 0x7f415c0abae0 con 0x7f415c0b17d0 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 >> v1:172.21.15.52:6790/0 conn(0x7f415c1422b0 legacy=0x7f415c13abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 >> v1:172.21.15.52:6789/0 conn(0x7f415c142690 legacy=0x7f415c13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 1 monclient: found mon.b 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c14dcc0 con 0x7f415c0b17d0 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:13.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4143fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 10 monclient: _renew_subs 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c14df70 con 0x7f415c0b17d0 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 10 monclient: _renew_subs 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c0ae560 con 0x7f415c0b17d0 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.758+0000 7f4161e0d640 1 librados: init done 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f414c0047e0 con 0x7f415c0b17d0 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f4143fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f414c004af0 con 0x7f415c0b17d0 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f4143fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f4143fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f4143fff640 10 monclient: dump: 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.759+0000 7f41597fa640 4 set_mon_vals no callback set 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.760+0000 7f4161e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.761+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f414c004c50 con 0x7f415c0b17d0 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.761+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f414c084a70 con 0x7f415c0b17d0 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.761+0000 7f4143fff640 10 client.4264.objecter ms_dispatch 0x7f415c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:13.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.761+0000 7f4143fff640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.761+0000 7f4143fff640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.761+0000 7f4143fff640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.762+0000 7f4161e0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.762+0000 7f4161e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.762+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c005080 con 0x7f415c0b17d0 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.762+0000 7f415affd640 10 monclient: get_auth_request con 0x7f414806ded0 auth_method 0 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.762+0000 7f4143fff640 10 client.4264.objecter ms_handle_connect 0x7f414806ded0 2024-03-29T11:21:13.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.769+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f414c050350 con 0x7f415c0b17d0 2024-03-29T11:21:13.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.769+0000 7f4143fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:13.769+0000 7f4143fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:14.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.143+0000 7f4161e0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:14.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.143+0000 7f4161e0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:14.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.143+0000 7f4161e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:14.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.143+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f413c005330 con 0x7f415c0b17d0 2024-03-29T11:21:14.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.144+0000 7f4143fff640 1 -- 172.21.15.52:0/3076848452 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f414c052280 con 0x7f415c0b17d0 2024-03-29T11:21:14.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.144+0000 7f4143fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:14.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.144+0000 7f4143fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:14.120 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:14.121 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:14.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.147+0000 7f4161e0d640 10 librados: watch_flush enter 2024-03-29T11:21:14.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 10 librados: watch_flush exit 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 >> v1:172.21.15.79:6816/407480584 conn(0x7f414806ded0 legacy=0x7f4148070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 10 monclient: shutdown 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 >> v1:172.21.15.79:6789/0 conn(0x7f415c0b17d0 legacy=0x7f415c0aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 shutdown_connections 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.148+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 >> 172.21.15.52:0/3076848452 conn(0x7f415c066a00 msgr2=0x7f415c13d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:14.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.149+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 shutdown_connections 2024-03-29T11:21:14.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.149+0000 7f4161e0d640 1 -- 172.21.15.52:0/3076848452 wait complete. 2024-03-29T11:21:14.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:14.149+0000 7f4161e0d640 1 librados: shutdown 2024-03-29T11:21:14.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:20.947 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.102+0000 7fc8cec84640 10 monclient: get_monmap_and_config 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.102+0000 7fc8cec84640 10 monclient: build_initial_monmap 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.102+0000 7fc8cec84640 10 monclient: monmap: 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:21.104368+0000 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:21.104368+0000 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:21.080 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:21.081 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:21.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.103+0000 7fc8cec84640 1 Processor -- start 2024-03-29T11:21:21.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.103+0000 7fc8cec84640 1 -- start start 2024-03-29T11:21:21.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.103+0000 7fc8cec84640 10 monclient: init 2024-03-29T11:21:21.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:21.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:21.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c80a9750 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:21.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c813e1e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:21.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c8071c30 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:21.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80ab300 con 0x7fc8c813e1e0 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cd221640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc8c8071c30 0x7fc8c80ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41392/0 (socket says 172.21.15.52:41392) 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cd221640 1 -- 172.21.15.52:0/3612622281 learned_addr learned my addr 172.21.15.52:0/3612622281 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80ac2f0 con 0x7fc8c80a9750 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80ad8d0 con 0x7fc8c8071c30 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.104+0000 7fc8cec84640 10 monclient(hunting): authenticate will time out at 1011.246012s 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.105+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917762947 0 0) 0x7fc8c80ad8d0 con 0x7fc8c8071c30 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.105+0000 7fc8c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.105+0000 7fc8c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.105+0000 7fc8c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.105+0000 7fc8c77fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:21:21.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.105+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0003520 con 0x7fc8c8071c30 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.106+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558492655 0 0) 0x7fc8b0003520 con 0x7fc8c8071c30 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.106+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 >> v1:172.21.15.79:6789/0 conn(0x7fc8c80a9750 legacy=0x7fc8c813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.106+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 >> v1:172.21.15.52:6789/0 conn(0x7fc8c813e1e0 legacy=0x7fc8c80ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 1 monclient: found mon.noname-b 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:21.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80ada60 con 0x7fc8c8071c30 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc8bc001690 con 0x7fc8c8071c30 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: dump: 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.086 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8cec84640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8cec84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8bc001860 con 0x7fc8c8071c30 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 1 -- 172.21.15.52:0/3612622281 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc8bc001cb0 con 0x7fc8c8071c30 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:21.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8c77fe640 10 monclient: dump: 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8cec84640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:21.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.108+0000 7fc8cec84640 4 set_mon_vals no callback set 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 10 monclient: shutdown 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 >> v1:172.21.15.52:6790/0 conn(0x7fc8c8071c30 legacy=0x7fc8c80ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 shutdown_connections 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 >> 172.21.15.52:0/3612622281 conn(0x7fc8c806b660 msgr2=0x7fc8c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 shutdown_connections 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.109+0000 7fc8cec84640 1 -- 172.21.15.52:0/3612622281 wait complete. 2024-03-29T11:21:21.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.110+0000 7fc8cec84640 10 monclient: build_initial_monmap 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.110+0000 7fc8cec84640 10 monclient: monmap: 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.110+0000 7fc8cec84640 1 librados: starting msgr at 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.110+0000 7fc8cec84640 1 librados: starting objecter 2024-03-29T11:21:21.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 1 Processor -- start 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 1 -- start start 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 1 librados: setting wanted keys 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 1 librados: calling monclient init 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 10 monclient: init 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.111+0000 7fc8cec84640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c8071c30 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cca20640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc8c8071c30 0x7fc8c81421e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42594/0 (socket says 172.21.15.52:42594) 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cca20640 1 -- 172.21.15.52:0/2871496803 learned_addr learned my addr 172.21.15.52:0/2871496803 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c80a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 client.?.objecter ms_handle_connect 0x7fc8c8071c30 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:21.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c813e1e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c806fc70 con 0x7fc8c80a9750 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c8071720 con 0x7fc8c813e1e0 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c8071420 con 0x7fc8c8071c30 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8cec84640 10 monclient(hunting): authenticate will time out at 1011.253386s 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473988816 0 0) 0x7fc8c8071420 con 0x7fc8c8071c30 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4004eb0 con 0x7fc8c8071c30 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 client.?.objecter ms_handle_connect 0x7fc8c813e1e0 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:21.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194999132 0 0) 0x7fc8c8071720 con 0x7fc8c813e1e0 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c8071420 con 0x7fc8c813e1e0 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675341038 0 0) 0x7fc8b4004eb0 con 0x7fc8c8071c30 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8c8071720 con 0x7fc8c8071c30 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc8c0001ce0 con 0x7fc8c8071c30 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:21.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.112+0000 7fc8c57fa640 10 monclient(hunting): dump: 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.113+0000 7fc8c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.113+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3204664729 0 0) 0x7fc8c8071720 con 0x7fc8c8071c30 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.113+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 >> v1:172.21.15.79:6789/0 conn(0x7fc8c813e1e0 legacy=0x7fc8c806f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.113+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 >> v1:172.21.15.52:6789/0 conn(0x7fc8c80a9750 legacy=0x7fc8c8135700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 1 monclient: found mon.c 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8070680 con 0x7fc8c8071c30 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:21.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8cec84640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8cec84640 10 monclient: _renew_subs 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8cec84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.115+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c8070960 con 0x7fc8c8071c30 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8cec84640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8cec84640 10 monclient: _renew_subs 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8cec84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c81e7220 con 0x7fc8c8071c30 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8cec84640 1 librados: init done 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c0003970 con 0x7fc8c8071c30 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc8c0003cc0 con 0x7fc8c8071c30 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:21.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c57fa640 10 monclient: dump: 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.116+0000 7fc8c6ffd640 4 set_mon_vals no callback set 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.117+0000 7fc8cec84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.117+0000 7fc8cec84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.117+0000 7fc8cec84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.118+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a8005080 con 0x7fc8c8071c30 2024-03-29T11:21:21.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.118+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fc8c0003e20 con 0x7fc8c8071c30 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.119+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fc8c0084dd0 con 0x7fc8c8071c30 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.119+0000 7fc8c57fa640 10 client.4289.objecter ms_dispatch 0x7fc8c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.119+0000 7fc8c57fa640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.119+0000 7fc8c57fa640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.119+0000 7fc8c7fff640 10 monclient: get_auth_request con 0x7fc8b406dbf0 auth_method 0 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.119+0000 7fc8c57fa640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:21:21.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.121+0000 7fc8c57fa640 10 client.4289.objecter ms_handle_connect 0x7fc8b406dbf0 2024-03-29T11:21:21.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.127+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8c00506b0 con 0x7fc8c8071c30 2024-03-29T11:21:21.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.127+0000 7fc8c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:21.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.127+0000 7fc8c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:21.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8cec84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:21.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8cec84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:21.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8cec84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:21.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a8005330 con 0x7fc8c8071c30 2024-03-29T11:21:21.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8c57fa640 1 -- 172.21.15.52:0/2871496803 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fc8c00525e0 con 0x7fc8c8071c30 2024-03-29T11:21:21.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:21.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.445+0000 7fc8c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:21.423 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:21.423 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:21.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 10 librados: watch_flush enter 2024-03-29T11:21:21.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 10 librados: watch_flush exit 2024-03-29T11:21:21.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:21:21.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:21:21.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 >> v1:172.21.15.79:6816/407480584 conn(0x7fc8b406dbf0 legacy=0x7fc8b40700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 10 monclient: shutdown 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 >> v1:172.21.15.52:6790/0 conn(0x7fc8c8071c30 legacy=0x7fc8c81421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 shutdown_connections 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 >> 172.21.15.52:0/2871496803 conn(0x7fc8c806b660 msgr2=0x7fc8c813abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.449+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 shutdown_connections 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.450+0000 7fc8cec84640 1 -- 172.21.15.52:0/2871496803 wait complete. 2024-03-29T11:21:21.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:21.450+0000 7fc8cec84640 1 librados: shutdown 2024-03-29T11:21:21.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:28.248 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:28.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.427+0000 7fb5dfdc4640 10 monclient: get_monmap_and_config 2024-03-29T11:21:28.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.427+0000 7fb5dfdc4640 10 monclient: build_initial_monmap 2024-03-29T11:21:28.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.428+0000 7fb5dfdc4640 10 monclient: monmap: 2024-03-29T11:21:28.404 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:28.404 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:28.428930+0000 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:28.428930+0000 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:28.405 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.429+0000 7fb5dfdc4640 1 Processor -- start 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.430+0000 7fb5dfdc4640 1 -- start start 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.430+0000 7fb5dfdc4640 10 monclient: init 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.430+0000 7fb5dfdc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.430+0000 7fb5dfdc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d80b4330 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 10 monclient(hunting): picked mon.noname-a con 0x7fb5d81470b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 10 monclient(hunting): picked mon.noname-c con 0x7fb5d81439b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d806f2a0 con 0x7fb5d81470b0 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d8146b90 con 0x7fb5d81439b0 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d80b2ca0 con 0x7fb5d80b4330 2024-03-29T11:21:28.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.431+0000 7fb5dfdc4640 10 monclient(hunting): authenticate will time out at 1018.571138s 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5ddb60640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb5d80b4330 0x7fb5d80b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42719/0 (socket says 172.21.15.52:42719) 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5ddb60640 1 -- 172.21.15.52:0/4241799236 learned_addr learned my addr 172.21.15.52:0/4241799236 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810616975 0 0) 0x7fb5d8146b90 con 0x7fb5d81439b0 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5dcb5e640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5dcb5e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5dcb5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5dcb5e640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:21:28.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.433+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c4003520 con 0x7fb5d81439b0 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191188715 0 0) 0x7fb5d80b2ca0 con 0x7fb5d80b4330 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8146b90 con 0x7fb5d80b4330 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579221773 0 0) 0x7fb5d8146b90 con 0x7fb5d80b4330 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 >> v1:172.21.15.79:6789/0 conn(0x7fb5d81439b0 legacy=0x7fb5d8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.434+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 >> v1:172.21.15.52:6789/0 conn(0x7fb5d81470b0 legacy=0x7fb5d80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: _finish_hunting 0 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 1 monclient: found mon.noname-b 2024-03-29T11:21:28.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d806e0c0 con 0x7fb5d80b4330 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: _finish_auth 0 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: _check_auth_tickets 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dfdc4640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dfdc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb5d00019f0 con 0x7fb5d80b4330 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: dump: 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:28.414 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dfdc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5d0001bc0 con 0x7fb5d80b4330 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dcb5e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dfdc4640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:28.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.436+0000 7fb5dfdc4640 4 set_mon_vals no callback set 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dcb5e640 1 -- 172.21.15.52:0/4241799236 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb5d0001f50 con 0x7fb5d80b4330 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dcb5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dcb5e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dcb5e640 10 monclient: dump: 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:28.417 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dfdc4640 10 monclient: shutdown 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dfdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/4241799236 >> v1:172.21.15.52:6790/0 conn(0x7fb5d80b4330 legacy=0x7fb5d80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/4241799236 shutdown_connections 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.437+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/4241799236 >> 172.21.15.52:0/4241799236 conn(0x7fb5d8066a00 msgr2=0x7fb5d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.438+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/4241799236 shutdown_connections 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.438+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/4241799236 wait complete. 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 10 monclient: build_initial_monmap 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 10 monclient: monmap: 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:28.418 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 1 librados: starting msgr at 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 1 librados: starting objecter 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 1 Processor -- start 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 1 -- start start 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 1 librados: setting wanted keys 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 1 librados: calling monclient init 2024-03-29T11:21:28.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.439+0000 7fb5dfdc4640 10 monclient: init 2024-03-29T11:21:28.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.440+0000 7fb5dfdc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:28.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.440+0000 7fb5dfdc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:28.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.440+0000 7fb5dfdc4640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d80b4330 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:28.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 10 monclient(hunting): picked mon.noname-c con 0x7fb5d81439b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:28.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 10 monclient(hunting): picked mon.noname-a con 0x7fb5d81470b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:28.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d81dec10 con 0x7fb5d81470b0 2024-03-29T11:21:28.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d81de8a0 con 0x7fb5d80b4330 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d8124e90 con 0x7fb5d81439b0 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5dfdc4640 10 monclient(hunting): authenticate will time out at 1018.581200s 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ddb60640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fb5d80b4330 0x7fb5d81df130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:60894/0 (socket says 172.21.15.52:60894) 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ddb60640 1 -- 172.21.15.52:0/2346699793 learned_addr learned my addr 172.21.15.52:0/2346699793 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 10 client.?.objecter ms_handle_connect 0x7fb5d80b4330 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422728778 0 0) 0x7fb5d81de8a0 con 0x7fb5d80b4330 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:21:28.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5bc004eb0 con 0x7fb5d80b4330 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.441+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3562616278 0 0) 0x7fb5bc004eb0 con 0x7fb5d80b4330 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5d81de8a0 con 0x7fb5d80b4330 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb5d0001fd0 con 0x7fb5d80b4330 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:28.423 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3493821227 0 0) 0x7fb5d81de8a0 con 0x7fb5d80b4330 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.442+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 >> v1:172.21.15.52:6790/0 conn(0x7fb5d81439b0 legacy=0x7fb5d8125a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.443+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 >> v1:172.21.15.52:6789/0 conn(0x7fb5d81470b0 legacy=0x7fb5d8141140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 1 monclient: found mon.b 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d8125020 con 0x7fb5d80b4330 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:28.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 10 monclient: _renew_subs 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d8126130 con 0x7fb5d80b4330 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 10 monclient: _renew_subs 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d8141a30 con 0x7fb5d80b4330 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.444+0000 7fb5dfdc4640 1 librados: init done 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5d0004e10 con 0x7fb5d80b4330 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb5d00051e0 con 0x7fb5d80b4330 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5cffff640 4 set_mon_vals no callback set 2024-03-29T11:21:28.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.445+0000 7fb5ce7fc640 10 monclient: dump: 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:28.426 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:28.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.447+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fb5d0006430 con 0x7fb5d80b4330 2024-03-29T11:21:28.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.447+0000 7fb5dd35f640 10 monclient: get_auth_request con 0x7fb5bc06dbe0 auth_method 0 2024-03-29T11:21:28.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.448+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fb5d0084fe0 con 0x7fb5d80b4330 2024-03-29T11:21:28.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.448+0000 7fb5ce7fc640 10 client.4273.objecter ms_dispatch 0x7fb5d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.448+0000 7fb5ce7fc640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.448+0000 7fb5ce7fc640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.448+0000 7fb5ce7fc640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.448+0000 7fb5ce7fc640 10 client.4273.objecter ms_handle_connect 0x7fb5bc06dbe0 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.449+0000 7fb5dfdc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.449+0000 7fb5dfdc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.449+0000 7fb5dfdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:28.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.449+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c0005080 con 0x7fb5d80b4330 2024-03-29T11:21:28.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.458+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5d00508c0 con 0x7fb5d80b4330 2024-03-29T11:21:28.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.458+0000 7fb5ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:28.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.458+0000 7fb5ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:28.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.793+0000 7fb5dfdc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.793+0000 7fb5dfdc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.793+0000 7fb5dfdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.793+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5c0005330 con 0x7fb5d80b4330 2024-03-29T11:21:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.796+0000 7fb5ce7fc640 1 -- 172.21.15.52:0/2346699793 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fb5d00527f0 con 0x7fb5d80b4330 2024-03-29T11:21:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.796+0000 7fb5ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.796+0000 7fb5ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:28.773 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:28.773 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:28.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 10 librados: watch_flush enter 2024-03-29T11:21:28.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 10 librados: watch_flush exit 2024-03-29T11:21:28.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T11:21:28.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 10 client.4273.objecter successfully canceled tick 2024-03-29T11:21:28.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 >> v1:172.21.15.79:6816/407480584 conn(0x7fb5bc06dbe0 legacy=0x7fb5bc0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 10 monclient: shutdown 2024-03-29T11:21:28.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:28.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.800+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 >> v1:172.21.15.79:6789/0 conn(0x7fb5d80b4330 legacy=0x7fb5d81df130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.801+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 shutdown_connections 2024-03-29T11:21:28.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.801+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 >> 172.21.15.52:0/2346699793 conn(0x7fb5d8066a00 msgr2=0x7fb5d8070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:28.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.802+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 shutdown_connections 2024-03-29T11:21:28.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.802+0000 7fb5dfdc4640 1 -- 172.21.15.52:0/2346699793 wait complete. 2024-03-29T11:21:28.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:28.802+0000 7fb5dfdc4640 1 librados: shutdown 2024-03-29T11:21:28.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:35.603 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.777+0000 7fcc6ca93640 10 monclient: get_monmap_and_config 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.777+0000 7fcc6ca93640 10 monclient: build_initial_monmap 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.777+0000 7fcc6ca93640 10 monclient: monmap: 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:35.778322+0000 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:35.778322+0000 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:35.754 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:35.755 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:35.755 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:35.755 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:35.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.778+0000 7fcc6ca93640 1 Processor -- start 2024-03-29T11:21:35.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.778+0000 7fcc6ca93640 1 -- start start 2024-03-29T11:21:35.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.778+0000 7fcc6ca93640 10 monclient: init 2024-03-29T11:21:35.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:35.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient(hunting): picked mon.noname-a con 0x7fcc680a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient(hunting): picked mon.noname-b con 0x7fcc6813e1e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient(hunting): picked mon.noname-c con 0x7fcc68071c30 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680ab300 con 0x7fcc680a9750 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680ac2f0 con 0x7fcc68071c30 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680ad8d0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.779+0000 7fcc6ca93640 10 monclient(hunting): authenticate will time out at 1025.919502s 2024-03-29T11:21:35.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.780+0000 7fcc66d9d640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fcc68071c30 0x7fcc680ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:58970/0 (socket says 172.21.15.52:58970) 2024-03-29T11:21:35.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.780+0000 7fcc66d9d640 1 -- 172.21.15.52:0/4035123534 learned_addr learned my addr 172.21.15.52:0/4035123534 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708824863 0 0) 0x7fcc680ac2f0 con 0x7fcc68071c30 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48003520 con 0x7fcc68071c30 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484343392 0 0) 0x7fcc680ab300 con 0x7fcc680a9750 2024-03-29T11:21:35.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.783+0000 7fcc65d9b640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:21:35.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.784+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc680ac2f0 con 0x7fcc680a9750 2024-03-29T11:21:35.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.784+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4118896594 0 0) 0x7fcc48003520 con 0x7fcc68071c30 2024-03-29T11:21:35.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.784+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 >> v1:172.21.15.52:6790/0 conn(0x7fcc6813e1e0 legacy=0x7fcc680ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 >> v1:172.21.15.52:6789/0 conn(0x7fcc680a9750 legacy=0x7fcc6813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 1 monclient: found mon.noname-c 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680ada60 con 0x7fcc68071c30 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:35.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcc60001a10 con 0x7fcc68071c30 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: dump: 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:35.764 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc6ca93640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc6ca93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc60001bf0 con 0x7fcc68071c30 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc6ca93640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc6ca93640 4 set_mon_vals no callback set 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.785+0000 7fcc65d9b640 1 -- 172.21.15.52:0/4035123534 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcc60002040 con 0x7fcc68071c30 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc65d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc65d9b640 10 monclient: dump: 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.765 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc6ca93640 10 monclient: shutdown 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc6ca93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc6ca93640 1 -- 172.21.15.52:0/4035123534 >> v1:172.21.15.79:6789/0 conn(0x7fcc68071c30 legacy=0x7fcc680ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc6ca93640 1 -- 172.21.15.52:0/4035123534 shutdown_connections 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.786+0000 7fcc6ca93640 1 -- 172.21.15.52:0/4035123534 >> 172.21.15.52:0/4035123534 conn(0x7fcc6806b660 msgr2=0x7fcc6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.787+0000 7fcc6ca93640 1 -- 172.21.15.52:0/4035123534 shutdown_connections 2024-03-29T11:21:35.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.787+0000 7fcc6ca93640 1 -- 172.21.15.52:0/4035123534 wait complete. 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.788+0000 7fcc6ca93640 10 monclient: build_initial_monmap 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.788+0000 7fcc6ca93640 10 monclient: monmap: 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:35.769 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.788+0000 7fcc6ca93640 1 librados: starting msgr at 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.788+0000 7fcc6ca93640 1 librados: starting objecter 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.788+0000 7fcc6ca93640 1 Processor -- start 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 1 -- start start 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 1 librados: setting wanted keys 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 1 librados: calling monclient init 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient: init 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient(hunting): picked mon.noname-b con 0x7fcc68071c30 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient(hunting): picked mon.noname-a con 0x7fcc680a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient(hunting): picked mon.noname-c con 0x7fcc6813e1e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:35.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc681cd590 con 0x7fcc680a9750 2024-03-29T11:21:35.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc681cc1d0 con 0x7fcc68071c30 2024-03-29T11:21:35.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc681cc6d0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:35.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.789+0000 7fcc6ca93640 10 monclient(hunting): authenticate will time out at 1025.929914s 2024-03-29T11:21:35.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.793+0000 7fcc6659c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fcc68071c30 0x7fcc681cb3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:44838/0 (socket says 172.21.15.52:44838) 2024-03-29T11:21:35.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.793+0000 7fcc6659c640 1 -- 172.21.15.52:0/2013410341 learned_addr learned my addr 172.21.15.52:0/2013410341 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.794+0000 7fcc5effd640 10 client.?.objecter ms_handle_connect 0x7fcc68071c30 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.794+0000 7fcc5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.794+0000 7fcc5effd640 10 client.?.objecter ms_handle_connect 0x7fcc6813e1e0 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.794+0000 7fcc5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.795+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12063576 0 0) 0x7fcc681cc6d0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.795+0000 7fcc5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.795+0000 7fcc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.795+0000 7fcc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.795+0000 7fcc5effd640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:21:35.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.795+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c004eb0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 client.?.objecter ms_handle_connect 0x7fcc680a9750 2024-03-29T11:21:35.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3740390437 0 0) 0x7fcc4c004eb0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc4c005300 con 0x7fcc6813e1e0 2024-03-29T11:21:35.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcc60001f00 con 0x7fcc6813e1e0 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): dump: 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:35.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591155328 0 0) 0x7fcc681cd590 con 0x7fcc680a9750 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c005850 con 0x7fcc680a9750 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3592954727 0 0) 0x7fcc4c005300 con 0x7fcc6813e1e0 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 >> v1:172.21.15.79:6789/0 conn(0x7fcc68071c30 legacy=0x7fcc681cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 >> v1:172.21.15.52:6789/0 conn(0x7fcc680a9750 legacy=0x7fcc681c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 monclient: found mon.c 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc681cd870 con 0x7fcc6813e1e0 2024-03-29T11:21:35.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.796+0000 7fcc5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 10 monclient: _renew_subs 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc681cbcb0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 10 monclient: _renew_subs 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc681c9240 con 0x7fcc6813e1e0 2024-03-29T11:21:35.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.798+0000 7fcc6ca93640 1 librados: init done 2024-03-29T11:21:35.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc600020e0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcc60001f00 con 0x7fcc6813e1e0 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc5effd640 10 monclient: dump: 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:35.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.799+0000 7fcc64d99640 4 set_mon_vals no callback set 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.800+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fcc6001aee0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.800+0000 7fcc6ca93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc6ca93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc6ca93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc40005080 con 0x7fcc6813e1e0 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fcc60084bd0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc5effd640 10 client.4295.objecter ms_dispatch 0x7fcc68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc5effd640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.801+0000 7fcc5effd640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.802+0000 7fcc5effd640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.802+0000 7fcc6659c640 10 monclient: get_auth_request con 0x7fcc4c06e360 auth_method 0 2024-03-29T11:21:35.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.802+0000 7fcc5effd640 10 client.4295.objecter ms_handle_connect 0x7fcc4c06e360 2024-03-29T11:21:35.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.813+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc600504b0 con 0x7fcc6813e1e0 2024-03-29T11:21:35.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.813+0000 7fcc5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:35.813+0000 7fcc5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:36.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.139+0000 7fcc6ca93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:36.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.139+0000 7fcc6ca93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:36.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.139+0000 7fcc6ca93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:21:36.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.139+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc40005330 con 0x7fcc6813e1e0 2024-03-29T11:21:36.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.140+0000 7fcc5effd640 1 -- 172.21.15.52:0/2013410341 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fcc600523e0 con 0x7fcc6813e1e0 2024-03-29T11:21:36.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.140+0000 7fcc5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:36.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.140+0000 7fcc5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:36.117 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:36.117 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 10 librados: watch_flush enter 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 10 librados: watch_flush exit 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 >> v1:172.21.15.79:6816/407480584 conn(0x7fcc4c06e360 legacy=0x7fcc4c070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 10 monclient: shutdown 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:36.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.143+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 >> v1:172.21.15.52:6790/0 conn(0x7fcc6813e1e0 legacy=0x7fcc681ee850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.144+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 shutdown_connections 2024-03-29T11:21:36.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.144+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 >> 172.21.15.52:0/2013410341 conn(0x7fcc6806b660 msgr2=0x7fcc6813ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:36.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.145+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 shutdown_connections 2024-03-29T11:21:36.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.145+0000 7fcc6ca93640 1 -- 172.21.15.52:0/2013410341 wait complete. 2024-03-29T11:21:36.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:36.145+0000 7fcc6ca93640 1 librados: shutdown 2024-03-29T11:21:36.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:42.944 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.118+0000 7f00f64b3640 10 monclient: get_monmap_and_config 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.119+0000 7f00f64b3640 10 monclient: build_initial_monmap 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.119+0000 7f00f64b3640 10 monclient: monmap: 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:43.119949+0000 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:43.119949+0000 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:43.096 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:43.097 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:43.097 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:43.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.120+0000 7f00f64b3640 1 Processor -- start 2024-03-29T11:21:43.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.120+0000 7f00f64b3640 1 -- start start 2024-03-29T11:21:43.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.120+0000 7f00f64b3640 10 monclient: init 2024-03-29T11:21:43.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.121+0000 7f00f64b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:43.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.121+0000 7f00f64b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:43.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.124+0000 7f00f64b3640 10 monclient(hunting): picked mon.noname-b con 0x7f00f00a9ac0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:43.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.125+0000 7f00f64b3640 10 monclient(hunting): picked mon.noname-c con 0x7f00f013ee90 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:43.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.125+0000 7f00f64b3640 10 monclient(hunting): picked mon.noname-a con 0x7f00f00af900 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:43.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00f64b3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f00afd10 con 0x7f00f00af900 2024-03-29T11:21:43.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00f64b3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f00af6f0 con 0x7f00f013ee90 2024-03-29T11:21:43.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00f64b3640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f00aac80 con 0x7f00f00a9ac0 2024-03-29T11:21:43.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00f64b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:43.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00f64b3640 10 monclient(hunting): authenticate will time out at 1033.266197s 2024-03-29T11:21:43.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00effff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f00f00a9ac0 0x7f00f00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43455/0 (socket says 172.21.15.52:43455) 2024-03-29T11:21:43.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.126+0000 7f00effff640 1 -- 172.21.15.52:0/1195527908 learned_addr learned my addr 172.21.15.52:0/1195527908 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:43.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.128+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351020437 0 0) 0x7f00f00aac80 con 0x7f00f00a9ac0 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.128+0000 7f00eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.128+0000 7f00eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.128+0000 7f00eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.128+0000 7f00eeffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.128+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00dc003520 con 0x7f00f00a9ac0 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486405943 0 0) 0x7f00f00afd10 con 0x7f00f00af900 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:43.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f00aac80 con 0x7f00f00af900 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307242636 0 0) 0x7f00dc003520 con 0x7f00f00a9ac0 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 >> v1:172.21.15.79:6789/0 conn(0x7f00f013ee90 legacy=0x7f00f013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 >> v1:172.21.15.52:6789/0 conn(0x7f00f00af900 legacy=0x7f00f0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 1 monclient: found mon.noname-b 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.129+0000 7f00eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:21:43.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f00aaf60 con 0x7f00f00a9ac0 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f00e8001a10 con 0x7f00f00a9ac0 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: dump: 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.109 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00e8001c30 con 0x7f00f00a9ac0 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 1 -- 172.21.15.52:0/1195527908 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f00e8001fc0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.130+0000 7f00eeffd640 10 monclient: dump: 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.110 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.111 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.111 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:43.111 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:43.111 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:43.111 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:43.111 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.132+0000 7f00f64b3640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.132+0000 7f00f64b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.132+0000 7f00f64b3640 4 set_mon_vals no callback set 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.133+0000 7f00f64b3640 10 monclient: shutdown 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.133+0000 7f00f64b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.133+0000 7f00f64b3640 1 -- 172.21.15.52:0/1195527908 >> v1:172.21.15.52:6790/0 conn(0x7f00f00a9ac0 legacy=0x7f00f00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.133+0000 7f00f64b3640 1 -- 172.21.15.52:0/1195527908 shutdown_connections 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.133+0000 7f00f64b3640 1 -- 172.21.15.52:0/1195527908 >> 172.21.15.52:0/1195527908 conn(0x7f00f00b35e0 msgr2=0x7f00f00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.134+0000 7f00f64b3640 1 -- 172.21.15.52:0/1195527908 shutdown_connections 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.134+0000 7f00f64b3640 1 -- 172.21.15.52:0/1195527908 wait complete. 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 10 monclient: build_initial_monmap 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 10 monclient: monmap: 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:43.113 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 1 librados: starting msgr at 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 1 librados: starting objecter 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 1 Processor -- start 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 1 -- start start 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 1 librados: setting wanted keys 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 1 librados: calling monclient init 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.135+0000 7f00f64b3640 10 monclient: init 2024-03-29T11:21:43.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.136+0000 7f00f64b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:43.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.136+0000 7f00f64b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:43.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.139+0000 7f00f64b3640 10 monclient(hunting): picked mon.noname-a con 0x7f00f00a9ac0 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:43.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.139+0000 7f00effff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f00f00a9ac0 0x7f00f006e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:51945/0 (socket says 172.21.15.52:51945) 2024-03-29T11:21:43.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.139+0000 7f00effff640 1 -- 172.21.15.52:0/3353168630 learned_addr learned my addr 172.21.15.52:0/3353168630 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:43.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.139+0000 7f00f64b3640 10 monclient(hunting): picked mon.noname-b con 0x7f00f00af900 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00f64b3640 10 monclient(hunting): picked mon.noname-c con 0x7f00f013ee90 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f0141ca0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f0141f10 con 0x7f00f00af900 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f013d590 con 0x7f00f013ee90 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00f64b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00f64b3640 10 monclient(hunting): authenticate will time out at 1033.282301s 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.141+0000 7f00ecff9640 10 client.?.objecter ms_handle_connect 0x7f00f00a9ac0 2024-03-29T11:21:43.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.142+0000 7f00ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:43.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 client.?.objecter ms_handle_connect 0x7f00f00af900 2024-03-29T11:21:43.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053692515 0 0) 0x7f00f0141ca0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d0004eb0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 client.?.objecter ms_handle_connect 0x7f00f013ee90 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2135108854 0 0) 0x7f00d0004eb0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f0141ca0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f00e8001f80 con 0x7f00f00a9ac0 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:43.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): dump: 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 34846299 0 0) 0x7f00f0141ca0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 >> v1:172.21.15.52:6790/0 conn(0x7f00f013ee90 legacy=0x7f00f01e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 >> v1:172.21.15.79:6789/0 conn(0x7f00f00af900 legacy=0x7f00f01406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 monclient: found mon.a 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f013d720 con 0x7f00f00a9ac0 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.149+0000 7f00ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00e8004f70 con 0x7f00f00a9ac0 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f00e8005280 con 0x7f00f00a9ac0 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ecff9640 10 monclient: dump: 2024-03-29T11:21:43.130 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00f64b3640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00ee7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00f64b3640 10 monclient: _renew_subs 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00f64b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.150+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f0142330 con 0x7f00f00a9ac0 2024-03-29T11:21:43.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.151+0000 7f00f64b3640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:43.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.151+0000 7f00f64b3640 10 monclient: _renew_subs 2024-03-29T11:21:43.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.151+0000 7f00f64b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:43.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.151+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f013fb20 con 0x7f00f00a9ac0 2024-03-29T11:21:43.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.151+0000 7f00f64b3640 1 librados: init done 2024-03-29T11:21:43.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.152+0000 7f00f64b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:43.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.152+0000 7f00f64b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:43.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.152+0000 7f00f64b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:43.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.152+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f00ae6c0 con 0x7f00f00a9ac0 2024-03-29T11:21:43.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.157+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f00e801b250 con 0x7f00f00a9ac0 2024-03-29T11:21:43.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.158+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f00e8084f50 con 0x7f00f00a9ac0 2024-03-29T11:21:43.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.158+0000 7f00ecff9640 10 client.4302.objecter ms_dispatch 0x7f00f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:43.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.158+0000 7f00ecff9640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:43.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.158+0000 7f00ecff9640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:43.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.158+0000 7f00ecff9640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:21:43.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.161+0000 7f00ef7fe640 10 monclient: get_auth_request con 0x7f00d006dd50 auth_method 0 2024-03-29T11:21:43.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.162+0000 7f00ecff9640 10 client.4302.objecter ms_handle_connect 0x7f00d006dd50 2024-03-29T11:21:43.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.164+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00e8050830 con 0x7f00f00a9ac0 2024-03-29T11:21:43.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.165+0000 7f00ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:43.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.165+0000 7f00ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:43.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.497+0000 7f00f64b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:43.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.497+0000 7f00f64b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:43.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.497+0000 7f00f64b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:43.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.497+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00f0075f10 con 0x7f00f00a9ac0 2024-03-29T11:21:43.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.498+0000 7f00ecff9640 1 -- 172.21.15.52:0/3353168630 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f00e8052760 con 0x7f00f00a9ac0 2024-03-29T11:21:43.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.498+0000 7f00ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:43.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.498+0000 7f00ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:43.475 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:43.475 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:43.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.502+0000 7f00f64b3640 10 librados: watch_flush enter 2024-03-29T11:21:43.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.502+0000 7f00f64b3640 10 librados: watch_flush exit 2024-03-29T11:21:43.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.502+0000 7f00f64b3640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:21:43.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.502+0000 7f00f64b3640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:21:43.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.503+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 >> v1:172.21.15.79:6816/407480584 conn(0x7f00d006dd50 legacy=0x7f00d0070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.503+0000 7f00f64b3640 10 monclient: shutdown 2024-03-29T11:21:43.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.503+0000 7f00f64b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:43.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.503+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 >> v1:172.21.15.52:6789/0 conn(0x7f00f00a9ac0 legacy=0x7f00f006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.503+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 shutdown_connections 2024-03-29T11:21:43.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.503+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 >> 172.21.15.52:0/3353168630 conn(0x7f00f00b35e0 msgr2=0x7f00f00ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:43.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.504+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 shutdown_connections 2024-03-29T11:21:43.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.506+0000 7f00f64b3640 1 -- 172.21.15.52:0/3353168630 wait complete. 2024-03-29T11:21:43.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:43.506+0000 7f00f64b3640 1 librados: shutdown 2024-03-29T11:21:43.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:50.308 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:50.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.493+0000 7f2bf2b8c640 10 monclient: get_monmap_and_config 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.493+0000 7f2bf2b8c640 10 monclient: build_initial_monmap 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.494+0000 7f2bf2b8c640 10 monclient: monmap: 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:50.494893+0000 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:50.494893+0000 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:50.471 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:50.472 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:50.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.496+0000 7f2bf2b8c640 1 Processor -- start 2024-03-29T11:21:50.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.497+0000 7f2bf2b8c640 1 -- start start 2024-03-29T11:21:50.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.497+0000 7f2bf2b8c640 10 monclient: init 2024-03-29T11:21:50.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.498+0000 7f2bf2b8c640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:50.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.498+0000 7f2bf2b8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:50.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.498+0000 7f2bf2b8c640 10 monclient(hunting): picked mon.noname-b con 0x7f2bec0ae250 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:50.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.498+0000 7f2bf2b8c640 10 monclient(hunting): picked mon.noname-a con 0x7f2bec13f970 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf2b8c640 10 monclient(hunting): picked mon.noname-c con 0x7f2bec142800 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf2b8c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0afe00 con 0x7f2bec13f970 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf2b8c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec142c10 con 0x7f2bec142800 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf2b8c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0afb20 con 0x7f2bec0ae250 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf2b8c640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf2b8c640 10 monclient(hunting): authenticate will time out at 1040.641828s 2024-03-29T11:21:50.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf1129640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f2bec142800 0x7f2bec14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:38289/0 (socket says 172.21.15.52:38289) 2024-03-29T11:21:50.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.501+0000 7f2bf1129640 1 -- 172.21.15.52:0/1598751524 learned_addr learned my addr 172.21.15.52:0/1598751524 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:50.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.502+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845901487 0 0) 0x7f2bec142c10 con 0x7f2bec142800 2024-03-29T11:21:50.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.502+0000 7f2beb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:50.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.502+0000 7f2beb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:50.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.502+0000 7f2beb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:50.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.502+0000 7f2beb7fe640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:21:50.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.502+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd4003520 con 0x7f2bec142800 2024-03-29T11:21:50.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.503+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1590968050 0 0) 0x7f2bd4003520 con 0x7f2bec142800 2024-03-29T11:21:50.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.503+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 >> v1:172.21.15.52:6790/0 conn(0x7f2bec0ae250 legacy=0x7f2bec13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:50.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.503+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 >> v1:172.21.15.52:6789/0 conn(0x7f2bec13f970 legacy=0x7f2bec1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec0aab80 con 0x7f2bec142800 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:50.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2be4001a10 con 0x7f2bec142800 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.506+0000 7f2beb7fe640 10 monclient: dump: 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2be4001c70 con 0x7f2bec142800 2024-03-29T11:21:50.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2beb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2beb7fe640 1 -- 172.21.15.52:0/1598751524 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2be40020c0 con 0x7f2bec142800 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2beb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2beb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2beb7fe640 10 monclient: dump: 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2bf2b8c640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:21:50.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2bf2b8c640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:50.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.507+0000 7f2bf2b8c640 4 set_mon_vals no callback set 2024-03-29T11:21:50.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.508+0000 7f2bf2b8c640 10 monclient: shutdown 2024-03-29T11:21:50.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.508+0000 7f2bf2b8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:50.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.508+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1598751524 >> v1:172.21.15.79:6789/0 conn(0x7f2bec142800 legacy=0x7f2bec14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.514+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1598751524 shutdown_connections 2024-03-29T11:21:50.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.514+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1598751524 >> 172.21.15.52:0/1598751524 conn(0x7f2bec0b35e0 msgr2=0x7f2bec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:50.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.517+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1598751524 shutdown_connections 2024-03-29T11:21:50.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.517+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1598751524 wait complete. 2024-03-29T11:21:50.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.517+0000 7f2bf2b8c640 10 monclient: build_initial_monmap 2024-03-29T11:21:50.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.517+0000 7f2bf2b8c640 10 monclient: monmap: 2024-03-29T11:21:50.494 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:50.494 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:50.494 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 1 librados: starting msgr at 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 1 librados: starting objecter 2024-03-29T11:21:50.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 1 Processor -- start 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 1 -- start start 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 1 librados: setting wanted keys 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 1 librados: calling monclient init 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.518+0000 7f2bf2b8c640 10 monclient: init 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient(hunting): picked mon.noname-a con 0x7f2bec0ae250 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient(hunting): picked mon.noname-b con 0x7f2bec13f970 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient(hunting): picked mon.noname-c con 0x7f2bec142800 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec1dfb00 con 0x7f2bec0ae250 2024-03-29T11:21:50.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec141710 con 0x7f2bec13f970 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0accd0 con 0x7f2bec142800 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.519+0000 7f2bf2b8c640 10 monclient(hunting): authenticate will time out at 1040.660043s 2024-03-29T11:21:50.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.523+0000 7f2bebfff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f2bec13f970 0x7f2bec1dece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:38429/0 (socket says 172.21.15.52:38429) 2024-03-29T11:21:50.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2bebfff640 1 -- 172.21.15.52:0/1768121837 learned_addr learned my addr 172.21.15.52:0/1768121837 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 10 client.?.objecter ms_handle_connect 0x7f2bec13f970 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149319539 0 0) 0x7f2bec141710 con 0x7f2bec13f970 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:21:50.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.524+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd8004ee0 con 0x7f2bec13f970 2024-03-29T11:21:50.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1207771659 0 0) 0x7f2bd8004ee0 con 0x7f2bec13f970 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bec141710 con 0x7f2bec13f970 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2bdc001960 con 0x7f2bec13f970 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 10 monclient(hunting): dump: 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:50.505 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1357641013 0 0) 0x7f2bec141710 con 0x7f2bec13f970 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 >> v1:172.21.15.52:6790/0 conn(0x7f2bec142800 legacy=0x7f2bec1df3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.528+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 >> v1:172.21.15.52:6789/0 conn(0x7f2bec0ae250 legacy=0x7f2bec141000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:50.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 1 monclient: found mon.b 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec0acfb0 con 0x7f2bec13f970 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bdc001c80 con 0x7f2bec13f970 2024-03-29T11:21:50.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2bdc003a40 con 0x7f2bec13f970 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.530+0000 7f2beaffd640 4 set_mon_vals no callback set 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.531+0000 7f2be97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.531+0000 7f2be97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.531+0000 7f2be97fa640 10 monclient: dump: 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:50.508 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 10 monclient: _renew_subs 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec0ad290 con 0x7f2bec13f970 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 10 monclient: _renew_subs 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec0ac930 con 0x7f2bec13f970 2024-03-29T11:21:50.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.532+0000 7f2bf2b8c640 1 librados: init done 2024-03-29T11:21:50.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.534+0000 7f2bf2b8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:50.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.535+0000 7f2bf2b8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:50.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.535+0000 7f2bf2b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.535+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec075f10 con 0x7f2bec13f970 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.536+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f2bdc003c20 con 0x7f2bec13f970 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.537+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f2bdc084be0 con 0x7f2bec13f970 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.537+0000 7f2be97fa640 10 client.4288.objecter ms_dispatch 0x7f2bec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.537+0000 7f2be97fa640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.537+0000 7f2be97fa640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.537+0000 7f2be97fa640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.537+0000 7f2bf0928640 10 monclient: get_auth_request con 0x7f2bd806db60 auth_method 0 2024-03-29T11:21:50.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.538+0000 7f2be97fa640 10 client.4288.objecter ms_handle_connect 0x7f2bd806db60 2024-03-29T11:21:50.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.544+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bdc0504c0 con 0x7f2bec13f970 2024-03-29T11:21:50.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.544+0000 7f2be97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:50.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.544+0000 7f2be97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:50.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.872+0000 7f2bf2b8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.872+0000 7f2bf2b8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.872+0000 7f2bf2b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:21:50.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.872+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bec11f3b0 con 0x7f2bec13f970 2024-03-29T11:21:50.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.875+0000 7f2be97fa640 1 -- 172.21.15.52:0/1768121837 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f2bdc0523f0 con 0x7f2bec13f970 2024-03-29T11:21:50.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.875+0000 7f2be97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.875+0000 7f2be97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:50.852 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:50.852 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:50.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 10 librados: watch_flush enter 2024-03-29T11:21:50.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 10 librados: watch_flush exit 2024-03-29T11:21:50.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 10 client.4288.objecter successfully canceled tick 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 >> v1:172.21.15.79:6816/407480584 conn(0x7f2bd806db60 legacy=0x7f2bd8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 10 monclient: shutdown 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 >> v1:172.21.15.79:6789/0 conn(0x7f2bec13f970 legacy=0x7f2bec1dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 shutdown_connections 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 >> 172.21.15.52:0/1768121837 conn(0x7f2bec0b35e0 msgr2=0x7f2bec0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:50.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.879+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 shutdown_connections 2024-03-29T11:21:50.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.880+0000 7f2bf2b8c640 1 -- 172.21.15.52:0/1768121837 wait complete. 2024-03-29T11:21:50.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:50.880+0000 7f2bf2b8c640 1 librados: shutdown 2024-03-29T11:21:50.873 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:57.680 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:57.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.855+0000 7ff1ad56b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:57.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.855+0000 7ff1ad56b640 10 monclient: build_initial_monmap 2024-03-29T11:21:57.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.855+0000 7ff1ad56b640 10 monclient: monmap: 2024-03-29T11:21:57.832 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:57.832 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:57.832 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:21:57.856652+0000 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:21:57.856652+0000 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:21:57.833 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.856+0000 7ff1ad56b640 1 Processor -- start 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.856+0000 7ff1ad56b640 1 -- start start 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.856+0000 7ff1ad56b640 10 monclient: init 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1a80b0770 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient(hunting): picked mon.noname-c con 0x7ff1a814a700 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:57.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1a80ac5a0 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:57.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a80aba70 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a80aecb0 con 0x7ff1a814a700 2024-03-29T11:21:57.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a80af150 con 0x7ff1a80b0770 2024-03-29T11:21:57.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:57.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.857+0000 7ff1ad56b640 10 monclient(hunting): authenticate will time out at 1047.997656s 2024-03-29T11:21:57.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.863+0000 7ff1a67fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7ff1a814a700 0x7ff1a80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:37779/0 (socket says 172.21.15.52:37779) 2024-03-29T11:21:57.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.863+0000 7ff1a67fc640 1 -- 172.21.15.52:0/3407049618 learned_addr learned my addr 172.21.15.52:0/3407049618 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:57.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.864+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000152821 0 0) 0x7ff1a80aba70 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.864+0000 7ff1a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.864+0000 7ff1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.864+0000 7ff1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.864+0000 7ff1a5ffb640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.864+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff190003520 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653841078 0 0) 0x7ff1a80aecb0 con 0x7ff1a814a700 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a80aba70 con 0x7ff1a814a700 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320601103 0 0) 0x7ff190003520 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 >> v1:172.21.15.52:6790/0 conn(0x7ff1a80b0770 legacy=0x7ff1a8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 >> v1:172.21.15.79:6789/0 conn(0x7ff1a814a700 legacy=0x7ff1a80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:57.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a80ac9d0 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff1a0001a10 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.865+0000 7ff1a5ffb640 10 monclient: dump: 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.845 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.866+0000 7ff1ad56b640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:21:57.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.866+0000 7ff1ad56b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1a0001b70 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1a5ffb640 1 -- 172.21.15.52:0/3407049618 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff1a0001f80 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1a5ffb640 10 monclient: dump: 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:57.847 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1ad56b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.867+0000 7ff1ad56b640 4 set_mon_vals no callback set 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.868+0000 7ff1ad56b640 10 monclient: shutdown 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.868+0000 7ff1ad56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.868+0000 7ff1ad56b640 1 -- 172.21.15.52:0/3407049618 >> v1:172.21.15.52:6789/0 conn(0x7ff1a80ac5a0 legacy=0x7ff1a8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.869+0000 7ff1ad56b640 1 -- 172.21.15.52:0/3407049618 shutdown_connections 2024-03-29T11:21:57.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.869+0000 7ff1ad56b640 1 -- 172.21.15.52:0/3407049618 >> 172.21.15.52:0/3407049618 conn(0x7ff1a806b660 msgr2=0x7ff1a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:57.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.870+0000 7ff1ad56b640 1 -- 172.21.15.52:0/3407049618 shutdown_connections 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.870+0000 7ff1ad56b640 1 -- 172.21.15.52:0/3407049618 wait complete. 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 10 monclient: build_initial_monmap 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 10 monclient: monmap: 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:21:57.850 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 1 librados: starting msgr at 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 1 librados: starting objecter 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 1 Processor -- start 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 1 -- start start 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 1 librados: setting wanted keys 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 1 librados: calling monclient init 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.871+0000 7ff1ad56b640 10 monclient: init 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.872+0000 7ff1ad56b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.872+0000 7ff1ad56b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:57.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.872+0000 7ff1ad56b640 10 monclient(hunting): picked mon.noname-c con 0x7ff1a80ac5a0 addr v1:172.21.15.52:6790/0 2024-03-29T11:21:57.858 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.872+0000 7ff1ad56b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1a80b0770 addr v1:172.21.15.79:6789/0 2024-03-29T11:21:57.858 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.873+0000 7ff1ad56b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1a814a700 addr v1:172.21.15.52:6789/0 2024-03-29T11:21:57.858 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.873+0000 7ff1ad56b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a8121da0 con 0x7ff1a814a700 2024-03-29T11:21:57.858 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.873+0000 7ff1ad56b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a81236d0 con 0x7ff1a80b0770 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.873+0000 7ff1ad56b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a8148570 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.873+0000 7ff1ad56b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.873+0000 7ff1ad56b640 10 monclient(hunting): authenticate will time out at 1048.013115s 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.878+0000 7ff1a77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7ff1a814a700 0x7ff1a8122230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:42829/0 (socket says 172.21.15.52:42829) 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.878+0000 7ff1a77fe640 1 -- 172.21.15.52:0/2692235240 learned_addr learned my addr 172.21.15.52:0/2692235240 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 10 client.?.objecter ms_handle_connect 0x7ff1a814a700 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222665201 0 0) 0x7ff1a8121da0 con 0x7ff1a814a700 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.879+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff194004eb0 con 0x7ff1a814a700 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 10 client.?.objecter ms_handle_connect 0x7ff1a80ac5a0 2024-03-29T11:21:57.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929456710 0 0) 0x7ff1a8148570 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:21:57.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.880+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8121da0 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 333529458 0 0) 0x7ff1a8121da0 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1a8148570 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff198001830 con 0x7ff1a80ac5a0 2024-03-29T11:21:57.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.882+0000 7ff18f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:57.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 client.?.objecter ms_handle_connect 0x7ff1a80b0770 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528078936 0 0) 0x7ff194004eb0 con 0x7ff1a814a700 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1a8121da0 con 0x7ff1a814a700 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff1a0001f40 con 0x7ff1a814a700 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.866 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 574967951 0 0) 0x7ff1a8121da0 con 0x7ff1a814a700 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 >> v1:172.21.15.52:6790/0 conn(0x7ff1a80ac5a0 legacy=0x7ff1a8147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 >> v1:172.21.15.79:6789/0 conn(0x7ff1a80b0770 legacy=0x7ff1a81e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 monclient: found mon.a 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8148700 con 0x7ff1a814a700 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:57.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1a0003c20 con 0x7ff1a814a700 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff1a0004030 con 0x7ff1a814a700 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.883+0000 7ff18f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff18f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff18f7fe640 10 monclient: dump: 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:21:57.868 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1a57fa640 4 set_mon_vals no callback set 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 10 monclient: _renew_subs 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a8148950 con 0x7ff1a814a700 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 10 monclient: _renew_subs 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a81e7f40 con 0x7ff1a814a700 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.884+0000 7ff1ad56b640 1 librados: init done 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.886+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7ff1a0004190 con 0x7ff1a814a700 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.887+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7ff1a0084e40 con 0x7ff1a814a700 2024-03-29T11:21:57.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.887+0000 7ff18f7fe640 10 client.4308.objecter ms_dispatch 0x7ff1a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:57.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.887+0000 7ff18f7fe640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:57.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.887+0000 7ff18f7fe640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:57.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.887+0000 7ff18f7fe640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:21:57.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.887+0000 7ff1a6ffd640 10 monclient: get_auth_request con 0x7ff19406df10 auth_method 0 2024-03-29T11:21:57.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.888+0000 7ff1ad56b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.888+0000 7ff1ad56b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.888+0000 7ff1ad56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.888+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a80ab560 con 0x7ff1a814a700 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.889+0000 7ff18f7fe640 10 client.4308.objecter ms_handle_connect 0x7ff19406df10 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.895+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1a0050780 con 0x7ff1a814a700 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.895+0000 7ff18f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:57.895+0000 7ff18f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.221+0000 7ff1ad56b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.221+0000 7ff1ad56b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.221+0000 7ff1ad56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:21:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.221+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1a8121f50 con 0x7ff1a814a700 2024-03-29T11:21:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.223+0000 7ff18f7fe640 1 -- 172.21.15.52:0/2692235240 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7ff1a8121f50 con 0x7ff1a814a700 2024-03-29T11:21:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.223+0000 7ff18f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.223+0000 7ff18f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:58.201 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:21:58.201 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.227+0000 7ff1ad56b640 10 librados: watch_flush enter 2024-03-29T11:21:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.229+0000 7ff1ad56b640 10 librados: watch_flush exit 2024-03-29T11:21:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.229+0000 7ff1ad56b640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:21:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.229+0000 7ff1ad56b640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:21:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 >> v1:172.21.15.79:6816/407480584 conn(0x7ff19406df10 legacy=0x7ff1940703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 10 monclient: shutdown 2024-03-29T11:21:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 >> v1:172.21.15.52:6789/0 conn(0x7ff1a814a700 legacy=0x7ff1a8122230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 shutdown_connections 2024-03-29T11:21:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 >> 172.21.15.52:0/2692235240 conn(0x7ff1a806b660 msgr2=0x7ff1a80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:58.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 shutdown_connections 2024-03-29T11:21:58.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 -- 172.21.15.52:0/2692235240 wait complete. 2024-03-29T11:21:58.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:21:58.231+0000 7ff1ad56b640 1 librados: shutdown 2024-03-29T11:21:58.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:05.034 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:05.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.214+0000 7fb6e350e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:05.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.214+0000 7fb6e350e640 10 monclient: build_initial_monmap 2024-03-29T11:22:05.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.214+0000 7fb6e350e640 10 monclient: monmap: 2024-03-29T11:22:05.191 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:05.191 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:05.191 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:05.215588+0000 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:05.215588+0000 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:05.192 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:05.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.218+0000 7fb6e350e640 1 Processor -- start 2024-03-29T11:22:05.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.218+0000 7fb6e350e640 1 -- start start 2024-03-29T11:22:05.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.218+0000 7fb6e350e640 10 monclient: init 2024-03-29T11:22:05.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.219+0000 7fb6e350e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:05.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.219+0000 7fb6e350e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:05.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.219+0000 7fb6e350e640 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc0b0860 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:05.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.219+0000 7fb6e350e640 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc13ef30 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.224+0000 7fb6e12aa640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fb6dc0b0860 0x7fb6dc13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:58201/0 (socket says 172.21.15.52:58201) 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e0aa9640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb6dc13ef30 0x7fb6dc0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:41337/0 (socket says 172.21.15.52:41337) 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e12aa640 1 -- 172.21.15.52:0/399932044 learned_addr learned my addr 172.21.15.52:0/399932044 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e350e640 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc0af8e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0b0c40 con 0x7fb6dc13ef30 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0aed10 con 0x7fb6dc0b0860 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0afd50 con 0x7fb6dc0af8e0 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e350e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:05.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.225+0000 7fb6e350e640 10 monclient(hunting): authenticate will time out at 1055.366041s 2024-03-29T11:22:05.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053841056 0 0) 0x7fb6dc0b0c40 con 0x7fb6dc13ef30 2024-03-29T11:22:05.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8003db0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132297202 0 0) 0x7fb6dc0afd50 con 0x7fb6dc0af8e0 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc0b0c40 con 0x7fb6dc0af8e0 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254546623 0 0) 0x7fb6c8003db0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 >> v1:172.21.15.52:6790/0 conn(0x7fb6dc0af8e0 legacy=0x7fb6dc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 >> v1:172.21.15.79:6789/0 conn(0x7fb6dc0b0860 legacy=0x7fb6dc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 monclient: found mon.noname-a 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0ab1b0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb6cc0015c0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.226+0000 7fb6d3fff640 10 monclient: dump: 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.208 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.227+0000 7fb6e350e640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.227+0000 7fb6e350e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6cc0017a0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6e350e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6e350e640 4 set_mon_vals no callback set 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6d3fff640 1 -- 172.21.15.52:0/399932044 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb6cc0015c0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.229+0000 7fb6d3fff640 10 monclient: dump: 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:05.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.230+0000 7fb6e350e640 10 monclient: shutdown 2024-03-29T11:22:05.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.230+0000 7fb6e350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:05.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.230+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 >> v1:172.21.15.52:6789/0 conn(0x7fb6dc13ef30 legacy=0x7fb6dc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.230+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 shutdown_connections 2024-03-29T11:22:05.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.230+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 >> 172.21.15.52:0/399932044 conn(0x7fb6dc06b660 msgr2=0x7fb6dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:05.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.231+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 shutdown_connections 2024-03-29T11:22:05.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.231+0000 7fb6e350e640 1 -- 172.21.15.52:0/399932044 wait complete. 2024-03-29T11:22:05.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.232+0000 7fb6e350e640 10 monclient: build_initial_monmap 2024-03-29T11:22:05.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.232+0000 7fb6e350e640 10 monclient: monmap: 2024-03-29T11:22:05.212 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:05.212 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.232+0000 7fb6e350e640 1 librados: starting msgr at 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.232+0000 7fb6e350e640 1 librados: starting objecter 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.233+0000 7fb6e350e640 1 Processor -- start 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.233+0000 7fb6e350e640 1 -- start start 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.233+0000 7fb6e350e640 1 librados: setting wanted keys 2024-03-29T11:22:05.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.233+0000 7fb6e350e640 1 librados: calling monclient init 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.233+0000 7fb6e350e640 10 monclient: init 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.234+0000 7fb6e350e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.234+0000 7fb6e350e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc0af8e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc0b0860 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc13ef30 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:05.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0b3af0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0b3d60 con 0x7fb6dc0af8e0 2024-03-29T11:22:05.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc1d2ea0 con 0x7fb6dc13ef30 2024-03-29T11:22:05.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:05.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.235+0000 7fb6e350e640 10 monclient(hunting): authenticate will time out at 1055.375694s 2024-03-29T11:22:05.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6e0aa9640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb6dc0b0860 0x7fb6dc1d33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:41559/0 (socket says 172.21.15.52:41559) 2024-03-29T11:22:05.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6e0aa9640 1 -- 172.21.15.52:0/662515931 learned_addr learned my addr 172.21.15.52:0/662515931 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:05.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 client.?.objecter ms_handle_connect 0x7fb6dc0b0860 2024-03-29T11:22:05.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:05.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57157419 0 0) 0x7fb6dc0b3af0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c0004eb0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1592090056 0 0) 0x7fb6c0004eb0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6dc0b3af0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb6cc0015c0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:05.219 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4162258569 0 0) 0x7fb6dc0b3af0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.239+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 >> v1:172.21.15.52:6790/0 conn(0x7fb6dc13ef30 legacy=0x7fb6dc0b2070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:05.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 >> v1:172.21.15.79:6789/0 conn(0x7fb6dc0af8e0 legacy=0x7fb6dc132810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 1 monclient: found mon.a 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc139c90 con 0x7fb6dc0b0860 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6cc004a90 con 0x7fb6dc0b0860 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb6cc004ee0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.241+0000 7fb6d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.243+0000 7fb6d1ffb640 10 monclient: dump: 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:05.223 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.243+0000 7fb6d2ffd640 4 set_mon_vals no callback set 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 10 monclient: _renew_subs 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc132f20 con 0x7fb6dc0b0860 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 10 monclient: _renew_subs 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:05.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc1d3cb0 con 0x7fb6dc0b0860 2024-03-29T11:22:05.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.244+0000 7fb6e350e640 1 librados: init done 2024-03-29T11:22:05.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.246+0000 7fb6e350e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:05.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.246+0000 7fb6e350e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:05.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.246+0000 7fb6e350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:05.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.246+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc1d3030 con 0x7fb6dc0b0860 2024-03-29T11:22:05.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.251+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fb6cc005040 con 0x7fb6dc0b0860 2024-03-29T11:22:05.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.251+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fb6cc084c20 con 0x7fb6dc0b0860 2024-03-29T11:22:05.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.251+0000 7fb6d1ffb640 10 client.4314.objecter ms_dispatch 0x7fb6dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:05.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.252+0000 7fb6d1ffb640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:05.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.252+0000 7fb6d1ffb640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:05.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.252+0000 7fb6d1ffb640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-29T11:22:05.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.252+0000 7fb6e12aa640 10 monclient: get_auth_request con 0x7fb6c006dc90 auth_method 0 2024-03-29T11:22:05.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.252+0000 7fb6d1ffb640 10 client.4314.objecter ms_handle_connect 0x7fb6c006dc90 2024-03-29T11:22:05.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.257+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6cc050500 con 0x7fb6dc0b0860 2024-03-29T11:22:05.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.257+0000 7fb6d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:05.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.257+0000 7fb6d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:05.563 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:05.563 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.584+0000 7fb6e350e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.584+0000 7fb6e350e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.584+0000 7fb6e350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.584+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6dc139e20 con 0x7fb6dc0b0860 2024-03-29T11:22:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.585+0000 7fb6d1ffb640 1 -- 172.21.15.52:0/662515931 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fb6cc050500 con 0x7fb6dc0b0860 2024-03-29T11:22:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.585+0000 7fb6d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.585+0000 7fb6d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.588+0000 7fb6e350e640 10 librados: watch_flush enter 2024-03-29T11:22:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 10 librados: watch_flush exit 2024-03-29T11:22:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-29T11:22:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 10 client.4314.objecter successfully canceled tick 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 >> v1:172.21.15.79:6816/407480584 conn(0x7fb6c006dc90 legacy=0x7fb6c0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 10 monclient: shutdown 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 >> v1:172.21.15.52:6789/0 conn(0x7fb6dc0b0860 legacy=0x7fb6dc1d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 shutdown_connections 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 >> 172.21.15.52:0/662515931 conn(0x7fb6dc06b660 msgr2=0x7fb6dc134ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 shutdown_connections 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 -- 172.21.15.52:0/662515931 wait complete. 2024-03-29T11:22:05.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:05.589+0000 7fb6e350e640 1 librados: shutdown 2024-03-29T11:22:05.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:12.390 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:12.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.560+0000 7f131a2f6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:12.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.560+0000 7f131a2f6640 10 monclient: build_initial_monmap 2024-03-29T11:22:12.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.560+0000 7f131a2f6640 10 monclient: monmap: 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:12.561372+0000 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:12.561372+0000 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:12.537 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.561+0000 7f131a2f6640 1 Processor -- start 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.561+0000 7f131a2f6640 1 -- start start 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.561+0000 7f131a2f6640 10 monclient: init 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient(hunting): picked mon.noname-a con 0x7f13141457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient(hunting): picked mon.noname-b con 0x7f131413f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient(hunting): picked mon.noname-c con 0x7f13140a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140aaef0 con 0x7f13141457f0 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1314145bd0 con 0x7f13140a9d70 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1314145eb0 con 0x7f131413f650 2024-03-29T11:22:12.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:12.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.562+0000 7f131a2f6640 10 monclient(hunting): authenticate will time out at 1062.702807s 2024-03-29T11:22:12.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.565+0000 7f13137fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f131413f650 0x7f13140b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:59462/0 (socket says 172.21.15.52:59462) 2024-03-29T11:22:12.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.565+0000 7f13137fe640 1 -- 172.21.15.52:0/779559467 learned_addr learned my addr 172.21.15.52:0/779559467 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:12.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997793256 0 0) 0x7f1314145bd0 con 0x7f13140a9d70 2024-03-29T11:22:12.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12fc003520 con 0x7f13140a9d70 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738930735 0 0) 0x7f12fc003520 con 0x7f13140a9d70 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 >> v1:172.21.15.52:6790/0 conn(0x7f131413f650 legacy=0x7f13140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.567+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 >> v1:172.21.15.52:6789/0 conn(0x7f13141457f0 legacy=0x7f131413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:12.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:12.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 1 monclient: found mon.noname-c 2024-03-29T11:22:12.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:12.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:22:12.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1314146340 con 0x7f13140a9d70 2024-03-29T11:22:12.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1304001690 con 0x7f13140a9d70 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: dump: 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:12.546 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f131a2f6640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f131a2f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1304001960 con 0x7f13140a9d70 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.568+0000 7f1312ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.569+0000 7f1312ffd640 1 -- 172.21.15.52:0/779559467 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1304001cf0 con 0x7f13140a9d70 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.569+0000 7f1312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.569+0000 7f1312ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.569+0000 7f1312ffd640 10 monclient: dump: 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.547 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:12.548 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:12.548 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:12.548 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:12.548 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:12.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.569+0000 7f131a2f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:12.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.569+0000 7f131a2f6640 4 set_mon_vals no callback set 2024-03-29T11:22:12.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 10 monclient: shutdown 2024-03-29T11:22:12.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:12.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 1 -- 172.21.15.52:0/779559467 >> v1:172.21.15.79:6789/0 conn(0x7f13140a9d70 legacy=0x7f131414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 1 -- 172.21.15.52:0/779559467 shutdown_connections 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 1 -- 172.21.15.52:0/779559467 >> 172.21.15.52:0/779559467 conn(0x7f1314066a00 msgr2=0x7f1314066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 1 -- 172.21.15.52:0/779559467 shutdown_connections 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.570+0000 7f131a2f6640 1 -- 172.21.15.52:0/779559467 wait complete. 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.571+0000 7f131a2f6640 10 monclient: build_initial_monmap 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.571+0000 7f131a2f6640 10 monclient: monmap: 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:12.550 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:12.551 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:12.551 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:12.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 1 librados: starting msgr at 2024-03-29T11:22:12.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 1 librados: starting objecter 2024-03-29T11:22:12.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 1 Processor -- start 2024-03-29T11:22:12.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 1 -- start start 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 1 librados: setting wanted keys 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 1 librados: calling monclient init 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.576+0000 7f131a2f6640 10 monclient: init 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient(hunting): picked mon.noname-a con 0x7f13140a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient(hunting): picked mon.noname-b con 0x7f131413f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient(hunting): picked mon.noname-c con 0x7f13141457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140acd40 con 0x7f13140a9d70 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140ad020 con 0x7f131413f650 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13141da6d0 con 0x7f13141457f0 2024-03-29T11:22:12.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:12.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.577+0000 7f131a2f6640 10 monclient(hunting): authenticate will time out at 1062.717710s 2024-03-29T11:22:12.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f13137fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f131413f650 0x7f1314137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:40399/0 (socket says 172.21.15.52:40399) 2024-03-29T11:22:12.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f13137fe640 1 -- 172.21.15.52:0/1501287436 learned_addr learned my addr 172.21.15.52:0/1501287436 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 client.?.objecter ms_handle_connect 0x7f13141457f0 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117044872 0 0) 0x7f13141da6d0 con 0x7f13141457f0 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1300004eb0 con 0x7f13141457f0 2024-03-29T11:22:12.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 client.?.objecter ms_handle_connect 0x7f13140a9d70 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.581+0000 7f1310ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 client.?.objecter ms_handle_connect 0x7f131413f650 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062629306 0 0) 0x7f13140acd40 con 0x7f13140a9d70 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13141da6d0 con 0x7f13140a9d70 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101772540 0 0) 0x7f13140ad020 con 0x7f131413f650 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13140acd40 con 0x7f131413f650 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094703070 0 0) 0x7f1300004eb0 con 0x7f13141457f0 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13140ad020 con 0x7f13141457f0 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1304001650 con 0x7f13141457f0 2024-03-29T11:22:12.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:12.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:12.562 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:12.562 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:12.562 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.562 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.582+0000 7f1310ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1018268330 0 0) 0x7f13140ad020 con 0x7f13141457f0 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 >> v1:172.21.15.79:6789/0 conn(0x7f131413f650 legacy=0x7f1314137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 >> v1:172.21.15.52:6789/0 conn(0x7f13140a9d70 legacy=0x7f13141dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 1 monclient: found mon.c 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:12.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13141db300 con 0x7f13141457f0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f1310ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 10 monclient: _renew_subs 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131413fda0 con 0x7f13141457f0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 10 monclient: _renew_subs 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1314142cd0 con 0x7f13141457f0 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.583+0000 7f131a2f6640 1 librados: init done 2024-03-29T11:22:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13040017b0 con 0x7f13141457f0 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f1310ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1304003b40 con 0x7f13141457f0 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f1310ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f1310ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f1310ff9640 10 monclient: dump: 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:12.565 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f13127fc640 4 set_mon_vals no callback set 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.584+0000 7f12fa7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.585+0000 7f12fa7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.585+0000 7f12fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:12.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.585+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13141da530 con 0x7f13141457f0 2024-03-29T11:22:12.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.587+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f1304019b10 con 0x7f13141457f0 2024-03-29T11:22:12.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.587+0000 7f1313fff640 10 monclient: get_auth_request con 0x7f130006da50 auth_method 0 2024-03-29T11:22:12.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.588+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f1304083930 con 0x7f13141457f0 2024-03-29T11:22:12.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.588+0000 7f1310ff9640 10 client.4310.objecter ms_dispatch 0x7f1314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:12.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.588+0000 7f1310ff9640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:12.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.588+0000 7f1310ff9640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:12.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.588+0000 7f1310ff9640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T11:22:12.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.588+0000 7f1310ff9640 10 client.4310.objecter ms_handle_connect 0x7f130006da50 2024-03-29T11:22:12.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.595+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13141da530 con 0x7f13141457f0 2024-03-29T11:22:12.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.595+0000 7f1310ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:12.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.595+0000 7f1310ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:12.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.948+0000 7f12fa7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.948+0000 7f12fa7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.948+0000 7f12fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:12.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.948+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13141da530 con 0x7f13141457f0 2024-03-29T11:22:12.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.949+0000 7f1310ff9640 1 -- 172.21.15.52:0/1501287436 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f13141da530 con 0x7f13141457f0 2024-03-29T11:22:12.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.949+0000 7f1310ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.949+0000 7f1310ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:12.925 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:12.926 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.952+0000 7f12fa7fc640 10 librados: watch_flush enter 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.952+0000 7f12fa7fc640 10 librados: watch_flush exit 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 10 client.4310.objecter successfully canceled tick 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 >> v1:172.21.15.79:6816/407480584 conn(0x7f130006da50 legacy=0x7f130006ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 10 monclient: shutdown 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 >> v1:172.21.15.52:6790/0 conn(0x7f13141457f0 legacy=0x7f13141e28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 shutdown_connections 2024-03-29T11:22:12.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 >> 172.21.15.52:0/1501287436 conn(0x7f1314066a00 msgr2=0x7f13140b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:12.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 shutdown_connections 2024-03-29T11:22:12.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 -- 172.21.15.52:0/1501287436 wait complete. 2024-03-29T11:22:12.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:12.953+0000 7f12fa7fc640 1 librados: shutdown 2024-03-29T11:22:12.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:19.752 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.920+0000 7f10a4874640 10 monclient: get_monmap_and_config 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.920+0000 7f10a4874640 10 monclient: build_initial_monmap 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.920+0000 7f10a4874640 10 monclient: monmap: 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:19.921503+0000 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:19.921503+0000 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:19.897 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:19.898 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.921+0000 7f10a4874640 1 Processor -- start 2024-03-29T11:22:19.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.921+0000 7f10a4874640 1 -- start start 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.921+0000 7f10a4874640 10 monclient: init 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient(hunting): picked mon.noname-c con 0x7f1098006e30 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient(hunting): picked mon.noname-b con 0x7f1098035e00 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient(hunting): picked mon.noname-a con 0x7f10980c9670 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1098035970 con 0x7f10980c9670 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1098036280 con 0x7f1098006e30 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1098035640 con 0x7f1098035e00 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.922+0000 7f10a4874640 10 monclient(hunting): authenticate will time out at 1070.062915s 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109ffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f10980c9670 0x7f10980d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55080/0 (socket says 172.21.15.52:55080) 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109ffff640 1 -- 172.21.15.52:0/3149943247 learned_addr learned my addr 172.21.15.52:0/3149943247 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996966086 0 0) 0x7f1098035970 con 0x7f10980c9670 2024-03-29T11:22:19.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109e7fc640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.923+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1090003520 con 0x7f10980c9670 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326892683 0 0) 0x7f1098035640 con 0x7f1098035e00 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1098035970 con 0x7f1098035e00 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788786336 0 0) 0x7f1098036280 con 0x7f1098006e30 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1098035640 con 0x7f1098006e30 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808071438 0 0) 0x7f1090003520 con 0x7f10980c9670 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 >> v1:172.21.15.52:6790/0 conn(0x7f1098035e00 legacy=0x7f10980c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 >> v1:172.21.15.79:6789/0 conn(0x7f1098006e30 legacy=0x7f10980cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098034e20 con 0x7f10980c9670 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1094001690 con 0x7f10980c9670 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f109e7fc640 10 monclient: dump: 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f10a4874640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:22:19.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.924+0000 7f10a4874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:19.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10940017f0 con 0x7f10980c9670 2024-03-29T11:22:19.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f109e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f109e7fc640 1 -- 172.21.15.52:0/3149943247 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1094001c00 con 0x7f10980c9670 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f109e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f109e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f109e7fc640 10 monclient: dump: 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:19.906 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f10a4874640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.925+0000 7f10a4874640 4 set_mon_vals no callback set 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 10 monclient: shutdown 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 1 -- 172.21.15.52:0/3149943247 >> v1:172.21.15.52:6789/0 conn(0x7f10980c9670 legacy=0x7f10980d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 1 -- 172.21.15.52:0/3149943247 shutdown_connections 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 1 -- 172.21.15.52:0/3149943247 >> 172.21.15.52:0/3149943247 conn(0x7f1098005c00 msgr2=0x7f1098006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 1 -- 172.21.15.52:0/3149943247 shutdown_connections 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.926+0000 7f10a4874640 1 -- 172.21.15.52:0/3149943247 wait complete. 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.929+0000 7f10a4874640 10 monclient: build_initial_monmap 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.929+0000 7f10a4874640 10 monclient: monmap: 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:19.907 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:19.908 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.930+0000 7f10a4874640 1 librados: starting msgr at 2024-03-29T11:22:19.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.930+0000 7f10a4874640 1 librados: starting objecter 2024-03-29T11:22:19.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.930+0000 7f10a4874640 1 Processor -- start 2024-03-29T11:22:19.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.931+0000 7f10a4874640 1 -- start start 2024-03-29T11:22:19.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.931+0000 7f10a4874640 1 librados: setting wanted keys 2024-03-29T11:22:19.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.931+0000 7f10a4874640 1 librados: calling monclient init 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.931+0000 7f10a4874640 10 monclient: init 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient(hunting): picked mon.noname-c con 0x7f1098006e30 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient(hunting): picked mon.noname-b con 0x7f1098035e00 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient(hunting): picked mon.noname-a con 0x7f10980c9670 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109802ef40 con 0x7f10980c9670 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109802d150 con 0x7f1098035e00 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109802d430 con 0x7f1098006e30 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f10a4874640 10 monclient(hunting): authenticate will time out at 1070.072478s 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f109ffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f10980c9670 0x7f10980c1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:56228/0 (socket says 172.21.15.52:56228) 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.932+0000 7f109ffff640 1 -- 172.21.15.52:0/1002938483 learned_addr learned my addr 172.21.15.52:0/1002938483 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:19.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 client.?.objecter ms_handle_connect 0x7f1098006e30 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 client.?.objecter ms_handle_connect 0x7f10980c9670 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035939606 0 0) 0x7f109802d430 con 0x7f1098006e30 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1084004eb0 con 0x7f1098006e30 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895632023 0 0) 0x7f109802ef40 con 0x7f10980c9670 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f109802d430 con 0x7f10980c9670 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3621891165 0 0) 0x7f1084004eb0 con 0x7f1098006e30 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f109802ef40 con 0x7f1098006e30 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f10a004fdd0 con 0x7f1098006e30 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:19.916 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137671563 0 0) 0x7f109802d430 con 0x7f10980c9670 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1084004eb0 con 0x7f10980c9670 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1094001b00 con 0x7f10980c9670 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.933+0000 7f108bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.917 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 10 client.?.objecter ms_handle_connect 0x7f1098035e00 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3897741369 0 0) 0x7f1084004eb0 con 0x7f10980c9670 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 >> v1:172.21.15.52:6790/0 conn(0x7f1098006e30 legacy=0x7f109802d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 >> v1:172.21.15.79:6789/0 conn(0x7f1098035e00 legacy=0x7f10980c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 1 monclient: found mon.a 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:19.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10980c2400 con 0x7f10980c9670 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f108bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 10 monclient: _renew_subs 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10980314a0 con 0x7f10980c9670 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 10 monclient: _renew_subs 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1098033cb0 con 0x7f10980c9670 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.934+0000 7f10a4874640 1 librados: init done 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1094001d20 con 0x7f10980c9670 2024-03-29T11:22:19.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f108bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1094004c30 con 0x7f10980c9670 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f108bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f108bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f108bfff640 10 monclient: dump: 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:19.920 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:19.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.935+0000 7f109d7fa640 4 set_mon_vals no callback set 2024-03-29T11:22:19.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.936+0000 7f10a4874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.936+0000 7f10a4874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.936+0000 7f10a4874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:19.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.936+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1080005080 con 0x7f10980c9670 2024-03-29T11:22:19.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.937+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f109401ac00 con 0x7f10980c9670 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.937+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f1094084a80 con 0x7f10980c9670 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.937+0000 7f108bfff640 10 client.4323.objecter ms_dispatch 0x7f10a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.937+0000 7f108bfff640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.937+0000 7f108bfff640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.938+0000 7f108bfff640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.938+0000 7f109f7fe640 10 monclient: get_auth_request con 0x7f108406dd70 auth_method 0 2024-03-29T11:22:19.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.938+0000 7f108bfff640 10 client.4323.objecter ms_handle_connect 0x7f108406dd70 2024-03-29T11:22:19.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.944+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1094050360 con 0x7f10980c9670 2024-03-29T11:22:19.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.944+0000 7f108bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:19.944+0000 7f108bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:20.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.320+0000 7f10a4874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:20.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.321+0000 7f10a4874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:20.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.321+0000 7f10a4874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:20.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.321+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1080005330 con 0x7f10980c9670 2024-03-29T11:22:20.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.321+0000 7f108bfff640 1 -- 172.21.15.52:0/1002938483 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f1094052290 con 0x7f10980c9670 2024-03-29T11:22:20.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.321+0000 7f108bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:20.298 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.321+0000 7f108bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:20.298 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:20.298 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:20.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 10 librados: watch_flush enter 2024-03-29T11:22:20.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 10 librados: watch_flush exit 2024-03-29T11:22:20.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T11:22:20.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 10 client.4323.objecter successfully canceled tick 2024-03-29T11:22:20.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 >> v1:172.21.15.79:6816/407480584 conn(0x7f108406dd70 legacy=0x7f1084070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:20.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 10 monclient: shutdown 2024-03-29T11:22:20.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:20.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.325+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 >> v1:172.21.15.52:6789/0 conn(0x7f10980c9670 legacy=0x7f10980c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:20.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.326+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 shutdown_connections 2024-03-29T11:22:20.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.326+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 >> 172.21.15.52:0/1002938483 conn(0x7f1098005c00 msgr2=0x7f10980c8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:20.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.326+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 shutdown_connections 2024-03-29T11:22:20.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.326+0000 7f10a4874640 1 -- 172.21.15.52:0/1002938483 wait complete. 2024-03-29T11:22:20.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:20.326+0000 7f10a4874640 1 librados: shutdown 2024-03-29T11:22:20.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:27.123 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:27.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.281+0000 7f3a6a006640 10 monclient: get_monmap_and_config 2024-03-29T11:22:27.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.281+0000 7f3a6a006640 10 monclient: build_initial_monmap 2024-03-29T11:22:27.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.281+0000 7f3a6a006640 10 monclient: monmap: 2024-03-29T11:22:27.258 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:27.258 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:27.282639+0000 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:27.282639+0000 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:27.259 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.283+0000 7f3a6a006640 1 Processor -- start 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.283+0000 7f3a6a006640 1 -- start start 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.283+0000 7f3a6a006640 10 monclient: init 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient(hunting): picked mon.noname-b con 0x7f3a640b0770 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient(hunting): picked mon.noname-a con 0x7f3a6414a700 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient(hunting): picked mon.noname-c con 0x7f3a640ac5a0 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640aba70 con 0x7f3a6414a700 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640aecb0 con 0x7f3a640ac5a0 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640af150 con 0x7f3a640b0770 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a6a006640 10 monclient(hunting): authenticate will time out at 1077.424210s 2024-03-29T11:22:27.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a637fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f3a640b0770 0x7f3a64149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49266/0 (socket says 172.21.15.52:49266) 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a637fe640 1 -- 172.21.15.52:0/4110951748 learned_addr learned my addr 172.21.15.52:0/4110951748 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321260545 0 0) 0x7f3a640af150 con 0x7f3a640b0770 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a627fc640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.284+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a50003520 con 0x7f3a640b0770 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.285+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221438708 0 0) 0x7f3a50003520 con 0x7f3a640b0770 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.285+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 >> v1:172.21.15.79:6789/0 conn(0x7f3a640ac5a0 legacy=0x7f3a64151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:27.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.285+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 >> v1:172.21.15.52:6789/0 conn(0x7f3a6414a700 legacy=0x7f3a640abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:27.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:27.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:27.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:27.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:22:27.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640ac9d0 con 0x7f3a640b0770 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3a54001a10 con 0x7f3a640b0770 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: dump: 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:27.269 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a6a006640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a6a006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a54001be0 con 0x7f3a640b0770 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 1 -- 172.21.15.52:0/4110951748 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3a54002030 con 0x7f3a640b0770 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a627fc640 10 monclient: dump: 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:27.270 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a6a006640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.286+0000 7f3a6a006640 4 set_mon_vals no callback set 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.287+0000 7f3a6a006640 10 monclient: shutdown 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.287+0000 7f3a6a006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.287+0000 7f3a6a006640 1 -- 172.21.15.52:0/4110951748 >> v1:172.21.15.52:6790/0 conn(0x7f3a640b0770 legacy=0x7f3a64149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.288+0000 7f3a6a006640 1 -- 172.21.15.52:0/4110951748 shutdown_connections 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.288+0000 7f3a6a006640 1 -- 172.21.15.52:0/4110951748 >> 172.21.15.52:0/4110951748 conn(0x7f3a6406b660 msgr2=0x7f3a6406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.288+0000 7f3a6a006640 1 -- 172.21.15.52:0/4110951748 shutdown_connections 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.288+0000 7f3a6a006640 1 -- 172.21.15.52:0/4110951748 wait complete. 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 10 monclient: build_initial_monmap 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 10 monclient: monmap: 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 1 librados: starting msgr at 2024-03-29T11:22:27.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 1 librados: starting objecter 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 1 Processor -- start 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 1 -- start start 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 1 librados: setting wanted keys 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 1 librados: calling monclient init 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.289+0000 7f3a6a006640 10 monclient: init 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.290+0000 7f3a6a006640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.290+0000 7f3a6a006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.290+0000 7f3a6a006640 10 monclient(hunting): picked mon.noname-a con 0x7f3a640ac5a0 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.290+0000 7f3a6a006640 10 monclient(hunting): picked mon.noname-b con 0x7f3a640b0770 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.290+0000 7f3a6a006640 10 monclient(hunting): picked mon.noname-c con 0x7f3a6414a700 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.290+0000 7f3a6a006640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a64127ca0 con 0x7f3a640ac5a0 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a6a006640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a641263e0 con 0x7f3a640b0770 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a6a006640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a641258b0 con 0x7f3a6414a700 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a6a006640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:27.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a6a006640 10 monclient(hunting): authenticate will time out at 1077.431105s 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a62ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f3a640b0770 0x7f3a64144ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:38150/0 (socket says 172.21.15.52:38150) 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a62ffd640 1 -- 172.21.15.52:0/1986807842 learned_addr learned my addr 172.21.15.52:0/1986807842 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 client.?.objecter ms_handle_connect 0x7f3a6414a700 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986359215 0 0) 0x7f3a641258b0 con 0x7f3a6414a700 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a44004eb0 con 0x7f3a6414a700 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 client.?.objecter ms_handle_connect 0x7f3a640b0770 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:27.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183552570 0 0) 0x7f3a44004eb0 con 0x7f3a6414a700 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a641258b0 con 0x7f3a6414a700 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3a5c0017f0 con 0x7f3a6414a700 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:27.275 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.291+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006534790 0 0) 0x7f3a641263e0 con 0x7f3a640b0770 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a44004eb0 con 0x7f3a640b0770 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1032250548 0 0) 0x7f3a641258b0 con 0x7f3a6414a700 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 >> v1:172.21.15.79:6789/0 conn(0x7f3a640b0770 legacy=0x7f3a64144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:27.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.292+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 >> v1:172.21.15.52:6789/0 conn(0x7f3a640ac5a0 legacy=0x7f3a641de750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 1 monclient: found mon.c 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a641260b0 con 0x7f3a6414a700 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:22:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 10 monclient: _renew_subs 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a641451e0 con 0x7f3a6414a700 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 10 monclient: _renew_subs 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a64125d40 con 0x7f3a6414a700 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a6a006640 1 librados: init done 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a5c0035f0 con 0x7f3a6414a700 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3a5c001d80 con 0x7f3a6414a700 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a5bfff640 10 monclient: dump: 2024-03-29T11:22:27.279 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.293+0000 7f3a61ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.295+0000 7f3a6a006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.295+0000 7f3a6a006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.295+0000 7f3a6a006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.295+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a48005080 con 0x7f3a6414a700 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.296+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f3a5c019770 con 0x7f3a6414a700 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.296+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f3a5c0837d0 con 0x7f3a6414a700 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.296+0000 7f3a5bfff640 10 client.4322.objecter ms_dispatch 0x7f3a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.296+0000 7f3a5bfff640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.296+0000 7f3a5bfff640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.296+0000 7f3a5bfff640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.298+0000 7f3a637fe640 10 monclient: get_auth_request con 0x7f3a4406de00 auth_method 0 2024-03-29T11:22:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.299+0000 7f3a5bfff640 10 client.4322.objecter ms_handle_connect 0x7f3a4406de00 2024-03-29T11:22:27.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.303+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a5c04eee0 con 0x7f3a6414a700 2024-03-29T11:22:27.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.303+0000 7f3a5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:27.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.303+0000 7f3a5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.613+0000 7f3a6a006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.613+0000 7f3a6a006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.613+0000 7f3a6a006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.613+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a48005330 con 0x7f3a6414a700 2024-03-29T11:22:27.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.615+0000 7f3a5bfff640 1 -- 172.21.15.52:0/1986807842 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f3a5c050fe0 con 0x7f3a6414a700 2024-03-29T11:22:27.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.615+0000 7f3a5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:27.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.615+0000 7f3a5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:27.592 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:27.592 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 10 librados: watch_flush enter 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 10 librados: watch_flush exit 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 >> v1:172.21.15.79:6816/407480584 conn(0x7f3a4406de00 legacy=0x7f3a440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 10 monclient: shutdown 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 >> v1:172.21.15.52:6790/0 conn(0x7f3a6414a700 legacy=0x7f3a641e6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 shutdown_connections 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 >> 172.21.15.52:0/1986807842 conn(0x7f3a6406b660 msgr2=0x7f3a640b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 shutdown_connections 2024-03-29T11:22:27.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 -- 172.21.15.52:0/1986807842 wait complete. 2024-03-29T11:22:27.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:27.618+0000 7f3a6a006640 1 librados: shutdown 2024-03-29T11:22:27.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:34.415 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:34.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.580+0000 7f299364a640 10 monclient: get_monmap_and_config 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.580+0000 7f299364a640 10 monclient: build_initial_monmap 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.580+0000 7f299364a640 10 monclient: monmap: 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:34.582606+0000 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:34.582606+0000 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:34.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.581+0000 7f299364a640 1 Processor -- start 2024-03-29T11:22:34.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.581+0000 7f299364a640 1 -- start start 2024-03-29T11:22:34.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.581+0000 7f299364a640 10 monclient: init 2024-03-29T11:22:34.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:34.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:34.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient(hunting): picked mon.noname-a con 0x7f298c0acf60 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:34.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient(hunting): picked mon.noname-b con 0x7f298c13ff40 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:34.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient(hunting): picked mon.noname-c con 0x7f298c1403f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:34.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c0b4310 con 0x7f298c0acf60 2024-03-29T11:22:34.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c140800 con 0x7f298c1403f0 2024-03-29T11:22:34.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c0aec00 con 0x7f298c13ff40 2024-03-29T11:22:34.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:34.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.582+0000 7f299364a640 10 monclient(hunting): authenticate will time out at 1084.723922s 2024-03-29T11:22:34.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.586+0000 7f29913e6640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f298c0acf60 0x7f298c139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:43673/0 (socket says 172.21.15.52:43673) 2024-03-29T11:22:34.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.586+0000 7f29913e6640 1 -- 172.21.15.52:0/1655182275 learned_addr learned my addr 172.21.15.52:0/1655182275 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:34.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.586+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771899749 0 0) 0x7f298c140800 con 0x7f298c1403f0 2024-03-29T11:22:34.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:34.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:34.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2978003520 con 0x7f298c1403f0 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825910905 0 0) 0x7f298c0b4310 con 0x7f298c0acf60 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c140800 con 0x7f298c0acf60 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593412482 0 0) 0x7f298c140800 con 0x7f298c0acf60 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 >> v1:172.21.15.52:6790/0 conn(0x7f298c13ff40 legacy=0x7f298c148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 >> v1:172.21.15.79:6789/0 conn(0x7f298c1403f0 legacy=0x7f298c14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 monclient: found mon.noname-a 2024-03-29T11:22:34.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c0aeee0 con 0x7f298c0acf60 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2988001670 con 0x7f298c0acf60 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.587+0000 7f2983fff640 10 monclient: dump: 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.569 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f299364a640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f299364a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29880017d0 con 0x7f298c0acf60 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f2983fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f2983fff640 1 -- 172.21.15.52:0/1655182275 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2988001be0 con 0x7f298c0acf60 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f2983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f2983fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f2983fff640 10 monclient: dump: 2024-03-29T11:22:34.570 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f299364a640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.588+0000 7f299364a640 4 set_mon_vals no callback set 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.589+0000 7f299364a640 10 monclient: shutdown 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.589+0000 7f299364a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.589+0000 7f299364a640 1 -- 172.21.15.52:0/1655182275 >> v1:172.21.15.52:6789/0 conn(0x7f298c0acf60 legacy=0x7f298c139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.589+0000 7f299364a640 1 -- 172.21.15.52:0/1655182275 shutdown_connections 2024-03-29T11:22:34.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.589+0000 7f299364a640 1 -- 172.21.15.52:0/1655182275 >> 172.21.15.52:0/1655182275 conn(0x7f298c0b35e0 msgr2=0x7f298c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:34.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.590+0000 7f299364a640 1 -- 172.21.15.52:0/1655182275 shutdown_connections 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.590+0000 7f299364a640 1 -- 172.21.15.52:0/1655182275 wait complete. 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.591+0000 7f299364a640 10 monclient: build_initial_monmap 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.591+0000 7f299364a640 10 monclient: monmap: 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:34.578 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.591+0000 7f299364a640 1 librados: starting msgr at 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.591+0000 7f299364a640 1 librados: starting objecter 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.591+0000 7f299364a640 1 Processor -- start 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.592+0000 7f299364a640 1 -- start start 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.592+0000 7f299364a640 1 librados: setting wanted keys 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.592+0000 7f299364a640 1 librados: calling monclient init 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.592+0000 7f299364a640 10 monclient: init 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient(hunting): picked mon.noname-b con 0x7f298c0acf60 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient(hunting): picked mon.noname-a con 0x7f298c13ff40 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient(hunting): picked mon.noname-c con 0x7f298c1403f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c13d620 con 0x7f298c13ff40 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c1e77d0 con 0x7f298c0acf60 2024-03-29T11:22:34.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c1e71d0 con 0x7f298c1403f0 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f299364a640 10 monclient(hunting): authenticate will time out at 1084.734648s 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f29913e6640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f298c0acf60 0x7f298c1e3cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:55692/0 (socket says 172.21.15.52:55692) 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.593+0000 7f29913e6640 1 -- 172.21.15.52:0/1415731083 learned_addr learned my addr 172.21.15.52:0/1415731083 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.594+0000 7f2981ffb640 10 client.?.objecter ms_handle_connect 0x7f298c0acf60 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.594+0000 7f2981ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448100241 0 0) 0x7f298c1e77d0 con 0x7f298c0acf60 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2970004eb0 con 0x7f298c0acf60 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 client.?.objecter ms_handle_connect 0x7f298c1403f0 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:34.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1737395290 0 0) 0x7f2970004eb0 con 0x7f298c0acf60 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f298c1e77d0 con 0x7f298c0acf60 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2988001ba0 con 0x7f298c0acf60 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:34.581 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.596+0000 7f2981ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095982774 0 0) 0x7f298c1e71d0 con 0x7f298c1403f0 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2970004eb0 con 0x7f298c1403f0 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3198413373 0 0) 0x7f298c1e77d0 con 0x7f298c0acf60 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 >> v1:172.21.15.52:6790/0 conn(0x7f298c1403f0 legacy=0x7f298c1ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 >> v1:172.21.15.52:6789/0 conn(0x7f298c13ff40 legacy=0x7f298c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 monclient: found mon.b 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:34.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c1e7360 con 0x7f298c0acf60 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.597+0000 7f2981ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 10 monclient: _renew_subs 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c13cb10 con 0x7f298c0acf60 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 10 monclient: _renew_subs 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c13ceb0 con 0x7f298c0acf60 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.598+0000 7f299364a640 1 librados: init done 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.599+0000 7f299364a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:34.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.599+0000 7f299364a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:34.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.599+0000 7f299364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:34.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.599+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c13a1f0 con 0x7f298c0acf60 2024-03-29T11:22:34.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.600+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2988004a90 con 0x7f298c0acf60 2024-03-29T11:22:34.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.600+0000 7f2981ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:34.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.600+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2988004ea0 con 0x7f298c0acf60 2024-03-29T11:22:34.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.600+0000 7f2981ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.600+0000 7f2981ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.600+0000 7f2981ffb640 10 monclient: dump: 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:34.586 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.602+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f2988005000 con 0x7f298c0acf60 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.602+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f2988084be0 con 0x7f298c0acf60 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.602+0000 7f2981ffb640 10 client.4309.objecter ms_dispatch 0x7f298c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.602+0000 7f2981ffb640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.602+0000 7f2981ffb640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.602+0000 7f2981ffb640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.603+0000 7f29837fe640 4 set_mon_vals no callback set 2024-03-29T11:22:34.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.604+0000 7f2990be5640 10 monclient: get_auth_request con 0x7f297006dd70 auth_method 0 2024-03-29T11:22:34.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.604+0000 7f2981ffb640 10 client.4309.objecter ms_handle_connect 0x7f297006dd70 2024-03-29T11:22:34.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.609+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29880504c0 con 0x7f298c0acf60 2024-03-29T11:22:34.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.609+0000 7f2981ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:34.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.609+0000 7f2981ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.927+0000 7f299364a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.928+0000 7f299364a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.928+0000 7f299364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:34.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.928+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f298c075f10 con 0x7f298c0acf60 2024-03-29T11:22:34.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.929+0000 7f2981ffb640 1 -- 172.21.15.52:0/1415731083 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f29880523f0 con 0x7f298c0acf60 2024-03-29T11:22:34.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.929+0000 7f2981ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.929+0000 7f2981ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:34.907 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:34.907 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.932+0000 7f299364a640 10 librados: watch_flush enter 2024-03-29T11:22:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.932+0000 7f299364a640 10 librados: watch_flush exit 2024-03-29T11:22:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.933+0000 7f299364a640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T11:22:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.933+0000 7f299364a640 10 client.4309.objecter successfully canceled tick 2024-03-29T11:22:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.933+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 >> v1:172.21.15.79:6816/407480584 conn(0x7f297006dd70 legacy=0x7f2970070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.933+0000 7f299364a640 10 monclient: shutdown 2024-03-29T11:22:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.933+0000 7f299364a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.933+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 >> v1:172.21.15.79:6789/0 conn(0x7f298c0acf60 legacy=0x7f298c1e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.934+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 shutdown_connections 2024-03-29T11:22:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.934+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 >> 172.21.15.52:0/1415731083 conn(0x7f298c0b35e0 msgr2=0x7f298c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:34.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.934+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 shutdown_connections 2024-03-29T11:22:34.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.934+0000 7f299364a640 1 -- 172.21.15.52:0/1415731083 wait complete. 2024-03-29T11:22:34.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:34.934+0000 7f299364a640 1 librados: shutdown 2024-03-29T11:22:34.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:41.735 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:41.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.909+0000 7f4ff9eaf640 10 monclient: get_monmap_and_config 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.909+0000 7f4ff9eaf640 10 monclient: build_initial_monmap 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.909+0000 7f4ff9eaf640 10 monclient: monmap: 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:41.910638+0000 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:41.910638+0000 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:41.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.910+0000 7f4ff9eaf640 1 Processor -- start 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.910+0000 7f4ff9eaf640 1 -- start start 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.910+0000 7f4ff9eaf640 10 monclient: init 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff41457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff413f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff40a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40aaef0 con 0x7f4ff413f650 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff4145bd0 con 0x7f4ff41457f0 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff4145eb0 con 0x7f4ff40a9d70 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:41.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.911+0000 7f4ff9eaf640 10 monclient(hunting): authenticate will time out at 1092.052003s 2024-03-29T11:22:41.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.912+0000 7f4ff37fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f4ff41457f0 0x7f4ff413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:42882/0 (socket says 172.21.15.52:42882) 2024-03-29T11:22:41.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.912+0000 7f4ff37fe640 1 -- 172.21.15.52:0/1396649980 learned_addr learned my addr 172.21.15.52:0/1396649980 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.913+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471704074 0 0) 0x7f4ff4145bd0 con 0x7f4ff41457f0 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.913+0000 7f4ff27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.913+0000 7f4ff27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.913+0000 7f4ff27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.913+0000 7f4ff27fc640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.913+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003520 con 0x7f4ff41457f0 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.914+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3333166627 0 0) 0x7f4fdc003520 con 0x7f4ff41457f0 2024-03-29T11:22:41.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.914+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 >> v1:172.21.15.52:6790/0 conn(0x7f4ff40a9d70 legacy=0x7f4ff414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 >> v1:172.21.15.52:6789/0 conn(0x7f4ff413f650 legacy=0x7f4ff40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 1 monclient: found mon.noname-c 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff4146340 con 0x7f4ff41457f0 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4fe8001670 con 0x7f4ff41457f0 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff9eaf640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:22:41.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff9eaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: dump: 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff9eaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe80018d0 con 0x7f4ff41457f0 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff9eaf640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff9eaf640 4 set_mon_vals no callback set 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 1 -- 172.21.15.52:0/1396649980 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4fe8001d20 con 0x7f4ff41457f0 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.917+0000 7f4ff27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.918+0000 7f4ff27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.918+0000 7f4ff27fc640 10 monclient: dump: 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.918+0000 7f4ff9eaf640 10 monclient: shutdown 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.918+0000 7f4ff9eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:41.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.918+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/1396649980 >> v1:172.21.15.79:6789/0 conn(0x7f4ff41457f0 legacy=0x7f4ff413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.920+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/1396649980 shutdown_connections 2024-03-29T11:22:41.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.920+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/1396649980 >> 172.21.15.52:0/1396649980 conn(0x7f4ff4066a00 msgr2=0x7f4ff4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:41.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.922+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/1396649980 shutdown_connections 2024-03-29T11:22:41.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.922+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/1396649980 wait complete. 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.923+0000 7f4ff9eaf640 10 monclient: build_initial_monmap 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.923+0000 7f4ff9eaf640 10 monclient: monmap: 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:41.901 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:41.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 1 librados: starting msgr at 2024-03-29T11:22:41.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 1 librados: starting objecter 2024-03-29T11:22:41.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 1 Processor -- start 2024-03-29T11:22:41.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 1 -- start start 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 1 librados: setting wanted keys 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 1 librados: calling monclient init 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.924+0000 7f4ff9eaf640 10 monclient: init 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff9eaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff9eaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff9eaf640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff40a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff9eaf640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff413f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:41.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff2ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f4ff413f650 0x7f4ff4141f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:43558/0 (socket says 172.21.15.52:43558) 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff9eaf640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff41457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff2ffd640 1 -- 172.21.15.52:0/807413901 learned_addr learned my addr 172.21.15.52:0/807413901 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.925+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff4141a90 con 0x7f4ff40a9d70 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.926+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff41409d0 con 0x7f4ff413f650 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.926+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff4140c40 con 0x7f4ff41457f0 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.926+0000 7f4ff9eaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.926+0000 7f4ff9eaf640 10 monclient(hunting): authenticate will time out at 1092.066110s 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.926+0000 7f4fdbfff640 10 client.?.objecter ms_handle_connect 0x7f4ff413f650 2024-03-29T11:22:41.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.926+0000 7f4fdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799243778 0 0) 0x7f4ff41409d0 con 0x7f4ff413f650 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003760 con 0x7f4ff413f650 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 10 client.?.objecter ms_handle_connect 0x7f4ff40a9d70 2024-03-29T11:22:41.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.928+0000 7f4fdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065771119 0 0) 0x7f4ff4141a90 con 0x7f4ff40a9d70 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff41409d0 con 0x7f4ff40a9d70 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074001102 0 0) 0x7f4fdc003760 con 0x7f4ff413f650 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ff4141a90 con 0x7f4ff413f650 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4fe4001960 con 0x7f4ff413f650 2024-03-29T11:22:41.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): dump: 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:41.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.929+0000 7f4fdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 client.?.objecter ms_handle_connect 0x7f4ff41457f0 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173456529 0 0) 0x7f4ff4140c40 con 0x7f4ff41457f0 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003760 con 0x7f4ff41457f0 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 468323152 0 0) 0x7f4ff4141a90 con 0x7f4ff413f650 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 >> v1:172.21.15.52:6790/0 conn(0x7f4ff41457f0 legacy=0x7f4ff4142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 >> v1:172.21.15.52:6789/0 conn(0x7f4ff40a9d70 legacy=0x7f4ff4144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 monclient: found mon.b 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff4140f20 con 0x7f4ff413f650 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.930+0000 7f4fdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 10 monclient: _renew_subs 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff41441b0 con 0x7f4ff413f650 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 10 monclient: _renew_subs 2024-03-29T11:22:41.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff41efac0 con 0x7f4ff413f650 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.931+0000 7f4ff9eaf640 1 librados: init done 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.932+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe4001ac0 con 0x7f4ff413f650 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4ff9eaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4ff9eaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4ff9eaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff4075f10 con 0x7f4ff413f650 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4fdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4fe40038c0 con 0x7f4ff413f650 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4fdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.933+0000 7f4fdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 10 monclient: dump: 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.917 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f4fe4003ae0 con 0x7f4ff413f650 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4ff37fe640 10 monclient: get_auth_request con 0x7f4fdc06e040 auth_method 0 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f4fe4084a90 con 0x7f4ff413f650 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 10 client.4315.objecter ms_dispatch 0x7f4ff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:41.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:41.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.934+0000 7f4fdbfff640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T11:22:41.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.935+0000 7f4fdbfff640 10 client.4315.objecter ms_handle_connect 0x7f4fdc06e040 2024-03-29T11:22:41.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.935+0000 7f4ff1ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:41.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.941+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fe40503f0 con 0x7f4ff413f650 2024-03-29T11:22:41.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.941+0000 7f4fdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:41.941+0000 7f4fdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:42.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.281+0000 7f4ff9eaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:42.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.281+0000 7f4ff9eaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:42.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.281+0000 7f4ff9eaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:22:42.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.281+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ff4144340 con 0x7f4ff413f650 2024-03-29T11:22:42.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.282+0000 7f4fdbfff640 1 -- 172.21.15.52:0/807413901 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f4fe4052320 con 0x7f4ff413f650 2024-03-29T11:22:42.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.282+0000 7f4fdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:42.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.282+0000 7f4fdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:42.261 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:42.261 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:42.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.286+0000 7f4ff9eaf640 10 librados: watch_flush enter 2024-03-29T11:22:42.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.286+0000 7f4ff9eaf640 10 librados: watch_flush exit 2024-03-29T11:22:42.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.286+0000 7f4ff9eaf640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T11:22:42.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.286+0000 7f4ff9eaf640 10 client.4315.objecter successfully canceled tick 2024-03-29T11:22:42.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.287+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 >> v1:172.21.15.79:6816/407480584 conn(0x7f4fdc06e040 legacy=0x7f4fdc070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:42.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.287+0000 7f4ff9eaf640 10 monclient: shutdown 2024-03-29T11:22:42.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.287+0000 7f4ff9eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:42.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.287+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 >> v1:172.21.15.79:6789/0 conn(0x7f4ff413f650 legacy=0x7f4ff4141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:42.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.288+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 shutdown_connections 2024-03-29T11:22:42.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.288+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 >> 172.21.15.52:0/807413901 conn(0x7f4ff4066a00 msgr2=0x7f4ff41374a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:42.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.288+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 shutdown_connections 2024-03-29T11:22:42.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.289+0000 7f4ff9eaf640 1 -- 172.21.15.52:0/807413901 wait complete. 2024-03-29T11:22:42.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:42.289+0000 7f4ff9eaf640 1 librados: shutdown 2024-03-29T11:22:42.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:49.091 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:49.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.266+0000 7f53436f1640 10 monclient: get_monmap_and_config 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.266+0000 7f53436f1640 10 monclient: build_initial_monmap 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.266+0000 7f53436f1640 10 monclient: monmap: 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:49.267318+0000 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:49.267318+0000 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:49.244 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:49.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.267+0000 7f53436f1640 1 Processor -- start 2024-03-29T11:22:49.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.267+0000 7f53436f1640 1 -- start start 2024-03-29T11:22:49.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.267+0000 7f53436f1640 10 monclient: init 2024-03-29T11:22:49.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.268+0000 7f53436f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:49.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.268+0000 7f53436f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:49.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.268+0000 7f53436f1640 10 monclient(hunting): picked mon.noname-a con 0x7f533c0b4710 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:49.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.268+0000 7f53436f1640 10 monclient(hunting): picked mon.noname-b con 0x7f533c0b3970 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:49.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f53436f1640 10 monclient(hunting): picked mon.noname-c con 0x7f533c13f390 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:49.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f53436f1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0af020 con 0x7f533c0b4710 2024-03-29T11:22:49.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f53436f1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0b0910 con 0x7f533c13f390 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f53436f1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0b0bf0 con 0x7f533c0b3970 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f53436f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f53436f1640 10 monclient(hunting): authenticate will time out at 1099.411612s 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f5341c8e640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f533c13f390 0x7f533c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:48865/0 (socket says 172.21.15.52:48865) 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.271+0000 7f5341c8e640 1 -- 172.21.15.52:0/1539540522 learned_addr learned my addr 172.21.15.52:0/1539540522 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.272+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228684519 0 0) 0x7f533c0b0910 con 0x7f533c13f390 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.272+0000 7f5333fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.272+0000 7f5333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.272+0000 7f5333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.272+0000 7f5333fff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:22:49.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.272+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5328003520 con 0x7f533c13f390 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502941062 0 0) 0x7f5328003520 con 0x7f533c13f390 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 >> v1:172.21.15.52:6790/0 conn(0x7f533c0b3970 legacy=0x7f533c0af4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 >> v1:172.21.15.52:6789/0 conn(0x7f533c0b4710 legacy=0x7f533c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 monclient: found mon.noname-c 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c0aeb00 con 0x7f533c13f390 2024-03-29T11:22:49.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5334001a10 con 0x7f533c13f390 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: dump: 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:49.255 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5334001ca0 con 0x7f533c13f390 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 1 -- 172.21.15.52:0/1539540522 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5334002030 con 0x7f533c13f390 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.276+0000 7f5333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.277+0000 7f5333fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.277+0000 7f5333fff640 10 monclient: dump: 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:49.256 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:49.257 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:49.257 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:49.257 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:49.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.277+0000 7f53436f1640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:22:49.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.277+0000 7f53436f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:49.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.277+0000 7f53436f1640 4 set_mon_vals no callback set 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.278+0000 7f53436f1640 10 monclient: shutdown 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.278+0000 7f53436f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.278+0000 7f53436f1640 1 -- 172.21.15.52:0/1539540522 >> v1:172.21.15.79:6789/0 conn(0x7f533c13f390 legacy=0x7f533c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.278+0000 7f53436f1640 1 -- 172.21.15.52:0/1539540522 shutdown_connections 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.278+0000 7f53436f1640 1 -- 172.21.15.52:0/1539540522 >> 172.21.15.52:0/1539540522 conn(0x7f533c066a00 msgr2=0x7f533c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.279+0000 7f53436f1640 1 -- 172.21.15.52:0/1539540522 shutdown_connections 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.279+0000 7f53436f1640 1 -- 172.21.15.52:0/1539540522 wait complete. 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.280+0000 7f53436f1640 10 monclient: build_initial_monmap 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.280+0000 7f53436f1640 10 monclient: monmap: 2024-03-29T11:22:49.259 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 1 librados: starting msgr at 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 1 librados: starting objecter 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 1 Processor -- start 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 1 -- start start 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 1 librados: setting wanted keys 2024-03-29T11:22:49.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 1 librados: calling monclient init 2024-03-29T11:22:49.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.281+0000 7f53436f1640 10 monclient: init 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.282+0000 7f53436f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.282+0000 7f53436f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 10 monclient(hunting): picked mon.noname-c con 0x7f533c0b3970 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 10 monclient(hunting): picked mon.noname-b con 0x7f533c13f390 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 10 monclient(hunting): picked mon.noname-a con 0x7f533c135e80 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c137fc0 con 0x7f533c135e80 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c136df0 con 0x7f533c13f390 2024-03-29T11:22:49.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c1370d0 con 0x7f533c0b3970 2024-03-29T11:22:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.284+0000 7f53436f1640 10 monclient(hunting): authenticate will time out at 1099.424255s 2024-03-29T11:22:49.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f534148d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f533c0b3970 0x7f533c141c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:33489/0 (socket says 172.21.15.52:33489) 2024-03-29T11:22:49.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f534148d640 1 -- 172.21.15.52:0/4000774475 learned_addr learned my addr 172.21.15.52:0/4000774475 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 client.?.objecter ms_handle_connect 0x7f533c0b3970 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 client.?.objecter ms_handle_connect 0x7f533c13f390 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864116293 0 0) 0x7f533c1370d0 con 0x7f533c0b3970 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.287+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5320004eb0 con 0x7f533c0b3970 2024-03-29T11:22:49.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 client.?.objecter ms_handle_connect 0x7f533c135e80 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2701304994 0 0) 0x7f5320004eb0 con 0x7f533c0b3970 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f533c1370d0 con 0x7f533c0b3970 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f53380018b0 con 0x7f533c0b3970 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:49.268 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178126933 0 0) 0x7f533c136df0 con 0x7f533c13f390 2024-03-29T11:22:49.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5320004eb0 con 0x7f533c13f390 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.288+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2670209017 0 0) 0x7f533c1370d0 con 0x7f533c0b3970 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 >> v1:172.21.15.79:6789/0 conn(0x7f533c13f390 legacy=0x7f533c1378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 >> v1:172.21.15.52:6789/0 conn(0x7f533c135e80 legacy=0x7f533c1e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 1 monclient: found mon.c 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c135980 con 0x7f533c0b3970 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:49.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:49.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f5331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:49.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f53436f1640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:22:49.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f53436f1640 10 monclient: _renew_subs 2024-03-29T11:22:49.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f53436f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:49.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.289+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c134d50 con 0x7f533c0b3970 2024-03-29T11:22:49.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.293+0000 7f53436f1640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:49.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.293+0000 7f53436f1640 10 monclient: _renew_subs 2024-03-29T11:22:49.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.293+0000 7f53436f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:49.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.293+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c1e3790 con 0x7f533c0b3970 2024-03-29T11:22:49.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.293+0000 7f53436f1640 1 librados: init done 2024-03-29T11:22:49.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53380035f0 con 0x7f533c0b3970 2024-03-29T11:22:49.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f5331ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5338003a40 con 0x7f533c0b3970 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f5331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f5331ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f5331ffb640 10 monclient: dump: 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:49.274 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f53436f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.295+0000 7f53337fe640 4 set_mon_vals no callback set 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.296+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f5338003c60 con 0x7f533c0b3970 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.296+0000 7f53436f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.296+0000 7f53436f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.296+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c0688a0 con 0x7f533c0b3970 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.297+0000 7f5340c8c640 10 monclient: get_auth_request con 0x7f532006db90 auth_method 0 2024-03-29T11:22:49.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.297+0000 7f5331ffb640 10 client.4337.objecter ms_handle_connect 0x7f532006db90 2024-03-29T11:22:49.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.300+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f5338054800 con 0x7f533c0b3970 2024-03-29T11:22:49.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.300+0000 7f5331ffb640 10 client.4337.objecter ms_dispatch 0x7f533c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:49.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.300+0000 7f5331ffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:49.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.300+0000 7f5331ffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:49.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.301+0000 7f5331ffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:22:49.283 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.306+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5338050370 con 0x7f533c0b3970 2024-03-29T11:22:49.283 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.306+0000 7f5331ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:49.283 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.306+0000 7f5331ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:49.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.641+0000 7f53436f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.641+0000 7f53436f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.641+0000 7f53436f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:22:49.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.641+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f533c075f10 con 0x7f533c0b3970 2024-03-29T11:22:49.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.642+0000 7f5331ffb640 1 -- 172.21.15.52:0/4000774475 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f5338050370 con 0x7f533c0b3970 2024-03-29T11:22:49.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.642+0000 7f5331ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.642+0000 7f5331ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:49.622 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:49.622 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:49.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.648+0000 7f53436f1640 10 librados: watch_flush enter 2024-03-29T11:22:49.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.649+0000 7f53436f1640 10 librados: watch_flush exit 2024-03-29T11:22:49.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.649+0000 7f53436f1640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:22:49.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.649+0000 7f53436f1640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:22:49.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.651+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 >> v1:172.21.15.79:6816/407480584 conn(0x7f532006db90 legacy=0x7f5320070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.651+0000 7f53436f1640 10 monclient: shutdown 2024-03-29T11:22:49.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.651+0000 7f53436f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:49.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.651+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 >> v1:172.21.15.52:6790/0 conn(0x7f533c0b3970 legacy=0x7f533c141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.653+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 shutdown_connections 2024-03-29T11:22:49.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.653+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 >> 172.21.15.52:0/4000774475 conn(0x7f533c066a00 msgr2=0x7f533c13c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:49.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.654+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 shutdown_connections 2024-03-29T11:22:49.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.654+0000 7f53436f1640 1 -- 172.21.15.52:0/4000774475 wait complete. 2024-03-29T11:22:49.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:49.654+0000 7f53436f1640 1 librados: shutdown 2024-03-29T11:22:49.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:56.459 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:56.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.635+0000 7efcb029d640 10 monclient: get_monmap_and_config 2024-03-29T11:22:56.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.635+0000 7efcb029d640 10 monclient: build_initial_monmap 2024-03-29T11:22:56.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.636+0000 7efcb029d640 10 monclient: monmap: 2024-03-29T11:22:56.613 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:56.613 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.613 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:22:56.636905+0000 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:22:56.636905+0000 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:22:56.614 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.616 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.639+0000 7efcb029d640 1 Processor -- start 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.641+0000 7efcb029d640 1 -- start start 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.641+0000 7efcb029d640 10 monclient: init 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.642+0000 7efcb029d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.642+0000 7efcb029d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.642+0000 7efcb029d640 10 monclient(hunting): picked mon.noname-c con 0x7efca81457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.642+0000 7efcb029d640 10 monclient(hunting): picked mon.noname-b con 0x7efca813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:56.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.643+0000 7efcb029d640 10 monclient(hunting): picked mon.noname-a con 0x7efca80a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:56.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.643+0000 7efcb029d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca80aaef0 con 0x7efca80a9d70 2024-03-29T11:22:56.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.643+0000 7efcb029d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8145bd0 con 0x7efca81457f0 2024-03-29T11:22:56.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.643+0000 7efcb029d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8145eb0 con 0x7efca813f650 2024-03-29T11:22:56.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.643+0000 7efcb029d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:56.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.644+0000 7efcb029d640 10 monclient(hunting): authenticate will time out at 1106.784071s 2024-03-29T11:22:56.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.646+0000 7efcae83a640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7efca80a9d70 0x7efca814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:35987/0 (socket says 172.21.15.52:35987) 2024-03-29T11:22:56.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.646+0000 7efcae83a640 1 -- 172.21.15.52:0/3070009731 learned_addr learned my addr 172.21.15.52:0/3070009731 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023856481 0 0) 0x7efca80aaef0 con 0x7efca80a9d70 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc94003520 con 0x7efca80a9d70 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727726535 0 0) 0x7efc94003520 con 0x7efca80a9d70 2024-03-29T11:22:56.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.648+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 >> v1:172.21.15.52:6790/0 conn(0x7efca813f650 legacy=0x7efca80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:56.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 >> v1:172.21.15.79:6789/0 conn(0x7efca81457f0 legacy=0x7efca813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:56.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: _finish_hunting 0 2024-03-29T11:22:56.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 1 monclient: found mon.noname-a 2024-03-29T11:22:56.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:56.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8146340 con 0x7efca80a9d70 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: _finish_auth 0 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: _check_auth_tickets 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efca0001a10 con 0x7efca80a9d70 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: dump: 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.630 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efca0001c70 con 0x7efca80a9d70 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 1 -- 172.21.15.52:0/3070009731 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efca00020c0 con 0x7efca80a9d70 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.650+0000 7efcad037640 10 monclient: dump: 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:56.631 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:56.632 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.652+0000 7efcb029d640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.652+0000 7efcb029d640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.652+0000 7efcb029d640 4 set_mon_vals no callback set 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 10 monclient: shutdown 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 1 -- 172.21.15.52:0/3070009731 >> v1:172.21.15.52:6789/0 conn(0x7efca80a9d70 legacy=0x7efca814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 1 -- 172.21.15.52:0/3070009731 shutdown_connections 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 1 -- 172.21.15.52:0/3070009731 >> 172.21.15.52:0/3070009731 conn(0x7efca8066a00 msgr2=0x7efca8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 1 -- 172.21.15.52:0/3070009731 shutdown_connections 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.653+0000 7efcb029d640 1 -- 172.21.15.52:0/3070009731 wait complete. 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.654+0000 7efcb029d640 10 monclient: build_initial_monmap 2024-03-29T11:22:56.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.654+0000 7efcb029d640 10 monclient: monmap: 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 1 librados: starting msgr at 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 1 librados: starting objecter 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 1 Processor -- start 2024-03-29T11:22:56.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 1 -- start start 2024-03-29T11:22:56.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 1 librados: setting wanted keys 2024-03-29T11:22:56.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 1 librados: calling monclient init 2024-03-29T11:22:56.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.655+0000 7efcb029d640 10 monclient: init 2024-03-29T11:22:56.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.656+0000 7efcb029d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:56.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.656+0000 7efcb029d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:56.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.656+0000 7efcb029d640 10 monclient(hunting): picked mon.noname-b con 0x7efca80a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:22:56.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.658+0000 7efcb029d640 10 monclient(hunting): picked mon.noname-c con 0x7efca813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcb029d640 10 monclient(hunting): picked mon.noname-a con 0x7efca81457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcb029d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca81450a0 con 0x7efca81457f0 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcb029d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8131300 con 0x7efca80a9d70 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcb029d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8144620 con 0x7efca813f650 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcae039640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7efca80a9d70 0x7efca8141a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:56217/0 (socket says 172.21.15.52:56217) 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcb029d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcae039640 1 -- 172.21.15.52:0/4105389590 learned_addr learned my addr 172.21.15.52:0/4105389590 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:22:56.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.659+0000 7efcb029d640 10 monclient(hunting): authenticate will time out at 1106.800017s 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 client.?.objecter ms_handle_connect 0x7efca80a9d70 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 client.?.objecter ms_handle_connect 0x7efca813f650 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 client.?.objecter ms_handle_connect 0x7efca81457f0 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343328852 0 0) 0x7efca8131300 con 0x7efca80a9d70 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8c004eb0 con 0x7efca80a9d70 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448915634 0 0) 0x7efca8144620 con 0x7efca813f650 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8131300 con 0x7efca813f650 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719376795 0 0) 0x7efca81450a0 con 0x7efca81457f0 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:56.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.663+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8144620 con 0x7efca81457f0 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.666+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763580012 0 0) 0x7efca8144620 con 0x7efca81457f0 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca81450a0 con 0x7efca81457f0 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117376285 0 0) 0x7efc8c004eb0 con 0x7efca80a9d70 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca8144620 con 0x7efca80a9d70 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 312024000 0 0) 0x7efca8131300 con 0x7efca813f650 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc8c004eb0 con 0x7efca813f650 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efca0001830 con 0x7efca81457f0 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): dump: 2024-03-29T11:22:56.646 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efca4003f60 con 0x7efca80a9d70 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): dump: 2024-03-29T11:22:56.647 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efc980016c0 con 0x7efca813f650 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:56.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient(hunting): dump: 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 220136788 0 0) 0x7efca81450a0 con 0x7efca81457f0 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 >> v1:172.21.15.52:6790/0 conn(0x7efca813f650 legacy=0x7efca813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 >> v1:172.21.15.79:6789/0 conn(0x7efca80a9d70 legacy=0x7efca8141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 monclient: found mon.a 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca81405b0 con 0x7efca81457f0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient: _finish_auth 0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efc9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efcb029d640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efcb029d640 10 monclient: _renew_subs 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efcb029d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.667+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca813c750 con 0x7efca81457f0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efcb029d640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efcb029d640 10 monclient: _renew_subs 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efcb029d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:56.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efca813df70 con 0x7efca81457f0 2024-03-29T11:22:56.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efcb029d640 1 librados: init done 2024-03-29T11:22:56.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efca0004ff0 con 0x7efca81457f0 2024-03-29T11:22:56.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efc9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:56.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.668+0000 7efcac836640 4 set_mon_vals no callback set 2024-03-29T11:22:56.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.670+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efca0001990 con 0x7efca81457f0 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.670+0000 7efc9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.670+0000 7efc9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.670+0000 7efc9effd640 10 monclient: dump: 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:22:56.653 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.670+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7efca001b170 con 0x7efca81457f0 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.671+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7efca0084d20 con 0x7efca81457f0 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.672+0000 7efcb029d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.672+0000 7efcb029d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.672+0000 7efcb029d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.672+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc90005080 con 0x7efca81457f0 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.672+0000 7efcae039640 10 monclient: get_auth_request con 0x7efc8c06e580 auth_method 0 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.672+0000 7efc9effd640 10 client.4338.objecter ms_dispatch 0x7efca8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.673+0000 7efc9effd640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.674+0000 7efc9effd640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.674+0000 7efc9effd640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:22:56.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.674+0000 7efc9effd640 10 client.4338.objecter ms_handle_connect 0x7efc8c06e580 2024-03-29T11:22:56.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.680+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efca0050680 con 0x7efca81457f0 2024-03-29T11:22:56.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.680+0000 7efc9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:56.680+0000 7efc9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:56.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.011+0000 7efcb029d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.011+0000 7efcb029d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.011+0000 7efcb029d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:22:56.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.011+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc90005330 con 0x7efca81457f0 2024-03-29T11:22:56.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.012+0000 7efc9effd640 1 -- 172.21.15.52:0/4105389590 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7efca00525b0 con 0x7efca81457f0 2024-03-29T11:22:56.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.012+0000 7efc9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.012+0000 7efc9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:56.990 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:22:56.990 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:56.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.016+0000 7efcb029d640 10 librados: watch_flush enter 2024-03-29T11:22:56.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.016+0000 7efcb029d640 10 librados: watch_flush exit 2024-03-29T11:22:56.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.016+0000 7efcb029d640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:22:56.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.016+0000 7efcb029d640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 >> v1:172.21.15.79:6816/407480584 conn(0x7efc8c06e580 legacy=0x7efc8c070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 10 monclient: shutdown 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 >> v1:172.21.15.52:6789/0 conn(0x7efca81457f0 legacy=0x7efca81e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 shutdown_connections 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 >> 172.21.15.52:0/4105389590 conn(0x7efca8066a00 msgr2=0x7efca80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 shutdown_connections 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.017+0000 7efcb029d640 1 -- 172.21.15.52:0/4105389590 wait complete. 2024-03-29T11:22:56.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:22:57.018+0000 7efcb029d640 1 librados: shutdown 2024-03-29T11:22:57.012 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:03.818 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:03.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.996+0000 7f4578486640 10 monclient: get_monmap_and_config 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.996+0000 7f4578486640 10 monclient: build_initial_monmap 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.996+0000 7f4578486640 10 monclient: monmap: 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:03.997558+0000 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:03.997558+0000 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:03.975 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:03.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.998+0000 7f4578486640 1 Processor -- start 2024-03-29T11:23:03.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.998+0000 7f4578486640 1 -- start start 2024-03-29T11:23:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.998+0000 7f4578486640 10 monclient: init 2024-03-29T11:23:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient(hunting): picked mon.noname-c con 0x7f45700b0770 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient(hunting): picked mon.noname-a con 0x7f457014a700 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient(hunting): picked mon.noname-b con 0x7f45700ac5a0 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700aba70 con 0x7f457014a700 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700aecb0 con 0x7f45700b0770 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700af150 con 0x7f45700ac5a0 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:03.999+0000 7f4578486640 10 monclient(hunting): authenticate will time out at 1114.139524s 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.003+0000 7f4576a23640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f45700ac5a0 0x7f4570151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:45619/0 (socket says 172.21.15.52:45619) 2024-03-29T11:23:03.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.003+0000 7f4576a23640 1 -- 172.21.15.52:0/34244229 learned_addr learned my addr 172.21.15.52:0/34244229 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:03.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567352631 0 0) 0x7f45700af150 con 0x7f45700ac5a0 2024-03-29T11:23:03.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:23:03.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c003520 con 0x7f45700ac5a0 2024-03-29T11:23:03.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361114769 0 0) 0x7f45700aba70 con 0x7f457014a700 2024-03-29T11:23:03.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45700af150 con 0x7f457014a700 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4128433090 0 0) 0x7f455c003520 con 0x7f45700ac5a0 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 >> v1:172.21.15.79:6789/0 conn(0x7f45700b0770 legacy=0x7f4570149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 >> v1:172.21.15.52:6789/0 conn(0x7f457014a700 legacy=0x7f45700abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: _finish_hunting 0 2024-03-29T11:23:03.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 monclient: found mon.noname-b 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45700ac9d0 con 0x7f45700ac5a0 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: _finish_auth 0 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: _check_auth_tickets 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4560001690 con 0x7f45700ac5a0 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.005+0000 7f4575220640 10 monclient: dump: 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:03.987 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.006+0000 7f4578486640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.006+0000 7f4578486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.006+0000 7f4575220640 1 -- 172.21.15.52:0/34244229 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45600017f0 con 0x7f45700ac5a0 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.006+0000 7f4575220640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.006+0000 7f4578486640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.006+0000 7f4578486640 4 set_mon_vals no callback set 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 10 monclient: shutdown 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 1 -- 172.21.15.52:0/34244229 >> v1:172.21.15.52:6790/0 conn(0x7f45700ac5a0 legacy=0x7f4570151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 1 -- 172.21.15.52:0/34244229 shutdown_connections 2024-03-29T11:23:03.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 1 -- 172.21.15.52:0/34244229 >> 172.21.15.52:0/34244229 conn(0x7f457006b660 msgr2=0x7f457006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:03.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 1 -- 172.21.15.52:0/34244229 shutdown_connections 2024-03-29T11:23:03.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.007+0000 7f4578486640 1 -- 172.21.15.52:0/34244229 wait complete. 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.009+0000 7f4578486640 10 monclient: build_initial_monmap 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.009+0000 7f4578486640 10 monclient: monmap: 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:03.991 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.009+0000 7f4578486640 1 librados: starting msgr at 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.009+0000 7f4578486640 1 librados: starting objecter 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.009+0000 7f4578486640 1 Processor -- start 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 1 -- start start 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 1 librados: setting wanted keys 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 1 librados: calling monclient init 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 10 monclient: init 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.010+0000 7f4578486640 10 monclient(hunting): picked mon.noname-a con 0x7f45700ac5a0 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 10 monclient(hunting): picked mon.noname-c con 0x7f45700b0770 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 10 monclient(hunting): picked mon.noname-b con 0x7f457014a700 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4570126280 con 0x7f45700ac5a0 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4570128470 con 0x7f457014a700 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4570145bb0 con 0x7f45700b0770 2024-03-29T11:23:03.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:03.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.011+0000 7f4578486640 10 monclient(hunting): authenticate will time out at 1114.151133s 2024-03-29T11:23:03.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4575a21640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f45700b0770 0x7f4570144b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:45765/0 (socket says 172.21.15.52:45765) 2024-03-29T11:23:03.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4575a21640 1 -- 172.21.15.52:0/4147349659 learned_addr learned my addr 172.21.15.52:0/4147349659 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 10 client.?.objecter ms_handle_connect 0x7f45700b0770 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816995096 0 0) 0x7f4570145bb0 con 0x7f45700b0770 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.016+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554004eb0 con 0x7f45700b0770 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833626451 0 0) 0x7f4554004eb0 con 0x7f45700b0770 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4570145bb0 con 0x7f45700b0770 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4568001c90 con 0x7f45700b0770 2024-03-29T11:23:03.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2920829409 0 0) 0x7f4570145bb0 con 0x7f45700b0770 2024-03-29T11:23:03.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 >> v1:172.21.15.79:6789/0 conn(0x7f457014a700 legacy=0x7f4570145260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.017+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 >> v1:172.21.15.52:6789/0 conn(0x7f45700ac5a0 legacy=0x7f4570126710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:03.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:03.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 1 monclient: found mon.c 2024-03-29T11:23:03.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:03.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:03.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570145970 con 0x7f45700b0770 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.020+0000 7f4566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.021+0000 7f4578486640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.021+0000 7f4578486640 10 monclient: _renew_subs 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.021+0000 7f4578486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.021+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4570125da0 con 0x7f45700b0770 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.022+0000 7f4578486640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:04.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.022+0000 7f4578486640 10 monclient: _renew_subs 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.022+0000 7f4578486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.022+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45701dc320 con 0x7f45700b0770 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.022+0000 7f4578486640 1 librados: init done 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4568001df0 con 0x7f45700b0770 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4566ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4568003c30 con 0x7f45700b0770 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4566ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4574a1f640 4 set_mon_vals no callback set 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.023+0000 7f4566ffd640 10 monclient: dump: 2024-03-29T11:23:04.001 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:04.002 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.024+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f4568003e50 con 0x7f45700b0770 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.024+0000 7f4578486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.024+0000 7f4578486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.025+0000 7f4578486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.025+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558005080 con 0x7f45700b0770 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.025+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f4568084e90 con 0x7f45700b0770 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.025+0000 7f4566ffd640 10 client.4346.objecter ms_dispatch 0x7f4570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.025+0000 7f4576222640 10 monclient: get_auth_request con 0x7f455406dc70 auth_method 0 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.026+0000 7f4566ffd640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.026+0000 7f4566ffd640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:04.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.026+0000 7f4566ffd640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:23:04.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.027+0000 7f4566ffd640 10 client.4346.objecter ms_handle_connect 0x7f455406dc70 2024-03-29T11:23:04.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.037+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4568050770 con 0x7f45700b0770 2024-03-29T11:23:04.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.037+0000 7f4566ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:04.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.037+0000 7f4566ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.370+0000 7f4578486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.370+0000 7f4578486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.370+0000 7f4578486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.370+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4558005330 con 0x7f45700b0770 2024-03-29T11:23:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.374+0000 7f4566ffd640 1 -- 172.21.15.52:0/4147349659 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f45680526a0 con 0x7f45700b0770 2024-03-29T11:23:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.374+0000 7f4566ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.374+0000 7f4566ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:04.355 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:04.355 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.380+0000 7f4578486640 10 librados: watch_flush enter 2024-03-29T11:23:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.380+0000 7f4578486640 10 librados: watch_flush exit 2024-03-29T11:23:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.380+0000 7f4578486640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:23:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.380+0000 7f4578486640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:23:04.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.381+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 >> v1:172.21.15.79:6816/407480584 conn(0x7f455406dc70 legacy=0x7f4554070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.381+0000 7f4578486640 10 monclient: shutdown 2024-03-29T11:23:04.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.381+0000 7f4578486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:04.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.381+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 >> v1:172.21.15.52:6790/0 conn(0x7f45700b0770 legacy=0x7f4570144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.382+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 shutdown_connections 2024-03-29T11:23:04.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.382+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 >> 172.21.15.52:0/4147349659 conn(0x7f457006b660 msgr2=0x7f45700b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:04.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.382+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 shutdown_connections 2024-03-29T11:23:04.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.382+0000 7f4578486640 1 -- 172.21.15.52:0/4147349659 wait complete. 2024-03-29T11:23:04.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:04.382+0000 7f4578486640 1 librados: shutdown 2024-03-29T11:23:04.377 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:11.184 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:11.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.369+0000 7fcae359e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:11.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.369+0000 7fcae359e640 10 monclient: build_initial_monmap 2024-03-29T11:23:11.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.369+0000 7fcae359e640 10 monclient: monmap: 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:11.370400+0000 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:11.370400+0000 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:11.348 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.370+0000 7fcae359e640 1 Processor -- start 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.370+0000 7fcae359e640 1 -- start start 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.370+0000 7fcae359e640 10 monclient: init 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient(hunting): picked mon.noname-a con 0x7fcae40b2800 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient(hunting): picked mon.noname-b con 0x7fcae40a9c00 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient(hunting): picked mon.noname-c con 0x7fcae40b38c0 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae4141eb0 con 0x7fcae40b2800 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae40b22d0 con 0x7fcae40b38c0 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae4142d30 con 0x7fcae40a9c00 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:11.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.371+0000 7fcae359e640 10 monclient(hunting): authenticate will time out at 1121.511965s 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae2d9d640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fcae40b38c0 0x7fcae414d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:33130/0 (socket says 172.21.15.52:33130) 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae2d9d640 1 -- 172.21.15.52:0/2702736437 learned_addr learned my addr 172.21.15.52:0/2702736437 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567511266 0 0) 0x7fcae40b22d0 con 0x7fcae40b38c0 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae159a640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:23:11.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.372+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcacc003520 con 0x7fcae40b38c0 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.373+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345060949 0 0) 0x7fcae4142d30 con 0x7fcae40a9c00 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.373+0000 7fcae159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.373+0000 7fcae159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.373+0000 7fcae159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae40b22d0 con 0x7fcae40a9c00 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928912385 0 0) 0x7fcacc003520 con 0x7fcae40b38c0 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 >> v1:172.21.15.52:6790/0 conn(0x7fcae40a9c00 legacy=0x7fcae413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 >> v1:172.21.15.52:6789/0 conn(0x7fcae40b2800 legacy=0x7fcae4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 monclient: found mon.noname-c 2024-03-29T11:23:11.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae40aaf90 con 0x7fcae40b38c0 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: _finish_auth 0 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae359e640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcadc001a10 con 0x7fcae40b38c0 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: dump: 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:11.354 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcadc001c30 con 0x7fcae40b38c0 2024-03-29T11:23:11.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.374+0000 7fcae359e640 4 set_mon_vals no callback set 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae159a640 1 -- 172.21.15.52:0/2702736437 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcadc001f40 con 0x7fcae40b38c0 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae159a640 10 monclient: dump: 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:11.356 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae359e640 10 monclient: shutdown 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.375+0000 7fcae359e640 1 -- 172.21.15.52:0/2702736437 >> v1:172.21.15.79:6789/0 conn(0x7fcae40b38c0 legacy=0x7fcae414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.376+0000 7fcae359e640 1 -- 172.21.15.52:0/2702736437 shutdown_connections 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.376+0000 7fcae359e640 1 -- 172.21.15.52:0/2702736437 >> 172.21.15.52:0/2702736437 conn(0x7fcae4066a00 msgr2=0x7fcae4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.376+0000 7fcae359e640 1 -- 172.21.15.52:0/2702736437 shutdown_connections 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.376+0000 7fcae359e640 1 -- 172.21.15.52:0/2702736437 wait complete. 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.377+0000 7fcae359e640 10 monclient: build_initial_monmap 2024-03-29T11:23:11.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.377+0000 7fcae359e640 10 monclient: monmap: 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 1 librados: starting msgr at 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 1 librados: starting objecter 2024-03-29T11:23:11.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 1 Processor -- start 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 1 -- start start 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 1 librados: setting wanted keys 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 1 librados: calling monclient init 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.378+0000 7fcae359e640 10 monclient: init 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient(hunting): picked mon.noname-c con 0x7fcae40a9c00 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:11.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient(hunting): picked mon.noname-b con 0x7fcae40b2800 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient(hunting): picked mon.noname-a con 0x7fcae40b38c0 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae406f9a0 con 0x7fcae40b38c0 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae4071fe0 con 0x7fcae40b2800 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae413f410 con 0x7fcae40a9c00 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.379+0000 7fcae359e640 10 monclient(hunting): authenticate will time out at 1121.519888s 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcae1d9b640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fcae40b2800 0x7fcae41e33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:36760/0 (socket says 172.21.15.52:36760) 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcae1d9b640 1 -- 172.21.15.52:0/1571016240 learned_addr learned my addr 172.21.15.52:0/1571016240 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 client.?.objecter ms_handle_connect 0x7fcae40b38c0 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 client.?.objecter ms_handle_connect 0x7fcae40b2800 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314393222 0 0) 0x7fcae4071fe0 con 0x7fcae40b2800 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.380+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac4004eb0 con 0x7fcae40b2800 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543447172 0 0) 0x7fcae406f9a0 con 0x7fcae40b38c0 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae4071fe0 con 0x7fcae40b38c0 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 122784342 0 0) 0x7fcac4004eb0 con 0x7fcae40b2800 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcae406f9a0 con 0x7fcae40b2800 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcad8001970 con 0x7fcae40b2800 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:11.364 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1093491545 0 0) 0x7fcae4071fe0 con 0x7fcae40b38c0 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcac4004eb0 con 0x7fcae40b38c0 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcadc001730 con 0x7fcae40b38c0 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.381+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3967731871 0 0) 0x7fcae406f9a0 con 0x7fcae40b2800 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.382+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 >> v1:172.21.15.52:6790/0 conn(0x7fcae40a9c00 legacy=0x7fcae413ca80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.382+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 >> v1:172.21.15.52:6789/0 conn(0x7fcae40b38c0 legacy=0x7fcae41e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:11.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 1 monclient: found mon.b 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae413d190 con 0x7fcae40b2800 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 10 monclient: _renew_subs 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae406fbd0 con 0x7fcae40b2800 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 10 monclient: _renew_subs 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae406f030 con 0x7fcae40b2800 2024-03-29T11:23:11.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.383+0000 7fcae359e640 1 librados: init done 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.384+0000 7fcae359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.384+0000 7fcae359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.384+0000 7fcae359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.384+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae406ffc0 con 0x7fcae40b2800 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.385+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcad8001ad0 con 0x7fcae40b2800 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fcad80038c0 con 0x7fcae40b2800 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcad2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcad2ffd640 10 monclient: dump: 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcae0d99640 4 set_mon_vals no callback set 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.386+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fcad8003aa0 con 0x7fcae40b2800 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fcad8084a20 con 0x7fcae40b2800 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcad2ffd640 10 client.4333.objecter ms_dispatch 0x7fcae4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcad2ffd640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcad2ffd640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcad2ffd640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcae259c640 10 monclient: get_auth_request con 0x7fcac406df10 auth_method 0 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.387+0000 7fcad2ffd640 10 client.4333.objecter ms_handle_connect 0x7fcac406df10 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.392+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcad8050380 con 0x7fcae40b2800 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.392+0000 7fcad2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.392+0000 7fcad2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.717+0000 7fcae359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.717+0000 7fcae359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.717+0000 7fcae359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:11.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.717+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcae404f730 con 0x7fcae40b2800 2024-03-29T11:23:11.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.719+0000 7fcad2ffd640 1 -- 172.21.15.52:0/1571016240 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fcad80522b0 con 0x7fcae40b2800 2024-03-29T11:23:11.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.719+0000 7fcad2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.719+0000 7fcad2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:11.698 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:11.698 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:11.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.723+0000 7fcae359e640 10 librados: watch_flush enter 2024-03-29T11:23:11.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.723+0000 7fcae359e640 10 librados: watch_flush exit 2024-03-29T11:23:11.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.723+0000 7fcae359e640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:23:11.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.723+0000 7fcae359e640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:23:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.725+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 >> v1:172.21.15.79:6816/407480584 conn(0x7fcac406df10 legacy=0x7fcac40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.725+0000 7fcae359e640 10 monclient: shutdown 2024-03-29T11:23:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.725+0000 7fcae359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.725+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 >> v1:172.21.15.79:6789/0 conn(0x7fcae40b2800 legacy=0x7fcae41e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.725+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 shutdown_connections 2024-03-29T11:23:11.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.725+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 >> 172.21.15.52:0/1571016240 conn(0x7fcae4066a00 msgr2=0x7fcae41404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:11.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.726+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 shutdown_connections 2024-03-29T11:23:11.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.726+0000 7fcae359e640 1 -- 172.21.15.52:0/1571016240 wait complete. 2024-03-29T11:23:11.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:11.726+0000 7fcae359e640 1 librados: shutdown 2024-03-29T11:23:11.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:18.528 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:18.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.706+0000 7f50854f4640 10 monclient: get_monmap_and_config 2024-03-29T11:23:18.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.706+0000 7f50854f4640 10 monclient: build_initial_monmap 2024-03-29T11:23:18.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.706+0000 7f50854f4640 10 monclient: monmap: 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:18.707844+0000 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:18.707844+0000 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:18.686 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.707+0000 7f50854f4640 1 Processor -- start 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 1 -- start start 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 10 monclient: init 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 10 monclient(hunting): picked mon.noname-c con 0x7f508006ade0 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 10 monclient(hunting): picked mon.noname-b con 0x7f508006ed60 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5080072340 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:18.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508006ff50 con 0x7f5080072340 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5080070180 con 0x7f508006ade0 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.708+0000 7f50854f4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508006f310 con 0x7f508006ed60 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f50854f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f50854f4640 10 monclient(hunting): authenticate will time out at 1128.849070s 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507effd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f508006ade0 0x7f50801499e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:53080/0 (socket says 172.21.15.52:53080) 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507effd640 1 -- 172.21.15.52:0/3527395048 learned_addr learned my addr 172.21.15.52:0/3527395048 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308911025 0 0) 0x7f5080070180 con 0x7f508006ade0 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507dffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:23:18.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.709+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068003520 con 0x7f508006ade0 2024-03-29T11:23:18.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421695531 0 0) 0x7f5068003520 con 0x7f508006ade0 2024-03-29T11:23:18.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 >> v1:172.21.15.52:6790/0 conn(0x7f508006ed60 legacy=0x7f5080071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:18.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 >> v1:172.21.15.52:6789/0 conn(0x7f5080072340 legacy=0x7f5080151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800aeed0 con 0x7f508006ade0 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5074001690 con 0x7f508006ade0 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: dump: 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:18.690 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f50854f4640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f50854f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 1 -- 172.21.15.52:0/3527395048 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5074001920 con 0x7f508006ade0 2024-03-29T11:23:18.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f507dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f50854f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.710+0000 7f50854f4640 4 set_mon_vals no callback set 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.711+0000 7f50854f4640 10 monclient: shutdown 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.711+0000 7f50854f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.711+0000 7f50854f4640 1 -- 172.21.15.52:0/3527395048 >> v1:172.21.15.79:6789/0 conn(0x7f508006ade0 legacy=0x7f50801499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.712+0000 7f50854f4640 1 -- 172.21.15.52:0/3527395048 shutdown_connections 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.712+0000 7f50854f4640 1 -- 172.21.15.52:0/3527395048 >> 172.21.15.52:0/3527395048 conn(0x7f50800b4930 msgr2=0x7f50800b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.712+0000 7f50854f4640 1 -- 172.21.15.52:0/3527395048 shutdown_connections 2024-03-29T11:23:18.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.712+0000 7f50854f4640 1 -- 172.21.15.52:0/3527395048 wait complete. 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.713+0000 7f50854f4640 10 monclient: build_initial_monmap 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.713+0000 7f50854f4640 10 monclient: monmap: 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:18.693 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 1 librados: starting msgr at 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 1 librados: starting objecter 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 1 Processor -- start 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 1 -- start start 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 1 librados: setting wanted keys 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 1 librados: calling monclient init 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.714+0000 7f50854f4640 10 monclient: init 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:18.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient(hunting): picked mon.noname-c con 0x7f508006ade0 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient(hunting): picked mon.noname-a con 0x7f508006ed60 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5080072340 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800a9f50 con 0x7f508006ed60 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800ab6b0 con 0x7f5080072340 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800aa3e0 con 0x7f508006ade0 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:18.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.715+0000 7f50854f4640 10 monclient(hunting): authenticate will time out at 1128.855995s 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f507e7fc640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f508006ed60 0x7f50801de5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:49054/0 (socket says 172.21.15.52:49054) 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f507e7fc640 1 -- 172.21.15.52:0/996362496 learned_addr learned my addr 172.21.15.52:0/996362496 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f50677fe640 10 client.?.objecter ms_handle_connect 0x7f508006ed60 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f50677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f50677fe640 10 client.?.objecter ms_handle_connect 0x7f5080072340 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f50677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f50677fe640 10 client.?.objecter ms_handle_connect 0x7f508006ade0 2024-03-29T11:23:18.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.716+0000 7f50677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457211516 0 0) 0x7f50800ab6b0 con 0x7f5080072340 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c004eb0 con 0x7f5080072340 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207802240 0 0) 0x7f50800aa3e0 con 0x7f508006ade0 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50800ab6b0 con 0x7f508006ade0 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3496506027 0 0) 0x7f506c004eb0 con 0x7f5080072340 2024-03-29T11:23:18.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.717+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50800aa3e0 con 0x7f5080072340 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5070001810 con 0x7f5080072340 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): dump: 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998297920 0 0) 0x7f50800a9f50 con 0x7f508006ed60 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c004eb0 con 0x7f508006ed60 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613065145 0 0) 0x7f50800ab6b0 con 0x7f508006ade0 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50800a9f50 con 0x7f508006ade0 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5074004940 con 0x7f508006ade0 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient(hunting): dump: 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2377433174 0 0) 0x7f50800aa3e0 con 0x7f5080072340 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 >> v1:172.21.15.52:6790/0 conn(0x7f508006ade0 legacy=0x7f50801413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 >> v1:172.21.15.52:6789/0 conn(0x7f508006ed60 legacy=0x7f50801de5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 monclient: found mon.b 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800aa670 con 0x7f5080072340 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.718+0000 7f50677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 10 monclient: _renew_subs 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5080141ad0 con 0x7f5080072340 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 10 monclient: _renew_subs 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50800ae6a0 con 0x7f5080072340 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50854f4640 1 librados: init done 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5070001d00 con 0x7f5080072340 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f50677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.719+0000 7f507d7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f5070003790 con 0x7f5080072340 2024-03-29T11:23:18.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50657fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50657fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5058005080 con 0x7f5080072340 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.720+0000 7f50677fe640 10 monclient: dump: 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:18.705 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:18.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.721+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f50700038f0 con 0x7f5080072340 2024-03-29T11:23:18.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f5070083750 con 0x7f5080072340 2024-03-29T11:23:18.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f50677fe640 10 client.4339.objecter ms_dispatch 0x7f5080000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f50677fe640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f50677fe640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f507effd640 10 monclient: get_auth_request con 0x7f506c06e0b0 auth_method 0 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f50677fe640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.722+0000 7f50677fe640 10 client.4339.objecter ms_handle_connect 0x7f506c06e0b0 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.727+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f507004f030 con 0x7f5080072340 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.727+0000 7f50677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:18.727+0000 7f50677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:19.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50657fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:19.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50657fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:19.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:19.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5058005330 con 0x7f5080072340 2024-03-29T11:23:19.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50677fe640 1 -- 172.21.15.52:0/996362496 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f5070050f60 con 0x7f5080072340 2024-03-29T11:23:19.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:19.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.161+0000 7f50677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:19.140 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:19.141 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:19.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 10 librados: watch_flush enter 2024-03-29T11:23:19.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 10 librados: watch_flush exit 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 >> v1:172.21.15.79:6816/407480584 conn(0x7f506c06e0b0 legacy=0x7f506c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 10 monclient: shutdown 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 >> v1:172.21.15.79:6789/0 conn(0x7f5080072340 legacy=0x7f50801e6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 shutdown_connections 2024-03-29T11:23:19.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.166+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 >> 172.21.15.52:0/996362496 conn(0x7f50800b4930 msgr2=0x7f50800b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:19.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.167+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 shutdown_connections 2024-03-29T11:23:19.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.167+0000 7f50657fa640 1 -- 172.21.15.52:0/996362496 wait complete. 2024-03-29T11:23:19.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:19.167+0000 7f50657fa640 1 librados: shutdown 2024-03-29T11:23:19.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:25.970 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:26.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.143+0000 7fa6db78c640 10 monclient: get_monmap_and_config 2024-03-29T11:23:26.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.143+0000 7fa6db78c640 10 monclient: build_initial_monmap 2024-03-29T11:23:26.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.143+0000 7fa6db78c640 10 monclient: monmap: 2024-03-29T11:23:26.122 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:26.122 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.122 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:26.144218+0000 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:26.144218+0000 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:26.123 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.144+0000 7fa6db78c640 1 Processor -- start 2024-03-29T11:23:26.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.144+0000 7fa6db78c640 1 -- start start 2024-03-29T11:23:26.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.144+0000 7fa6db78c640 10 monclient: init 2024-03-29T11:23:26.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:26.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:26.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d40b0500 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:26.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d40b08e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d40aeca0 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d4140620 con 0x7fa6d40aeca0 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d413daf0 con 0x7fa6d40b08e0 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40ae180 con 0x7fa6d40b0500 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6db78c640 10 monclient(hunting): authenticate will time out at 1136.285666s 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6d9528640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa6d40b0500 0x7fa6d413ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50812/0 (socket says 172.21.15.52:50812) 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.145+0000 7fa6d9528640 1 -- 172.21.15.52:0/1557219223 learned_addr learned my addr 172.21.15.52:0/1557219223 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207708852 0 0) 0x7fa6d413daf0 con 0x7fa6d40b08e0 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0003520 con 0x7fa6d40b08e0 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226006862 0 0) 0x7fa6d40ae180 con 0x7fa6d40b0500 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d413daf0 con 0x7fa6d40b0500 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776147957 0 0) 0x7fa6d4140620 con 0x7fa6d40aeca0 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d40ae180 con 0x7fa6d40aeca0 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641625265 0 0) 0x7fa6c0003520 con 0x7fa6d40b08e0 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 >> v1:172.21.15.52:6790/0 conn(0x7fa6d40b0500 legacy=0x7fa6d413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 >> v1:172.21.15.52:6789/0 conn(0x7fa6d40aeca0 legacy=0x7fa6d414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 monclient: found mon.noname-c 2024-03-29T11:23:26.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40accf0 con 0x7fa6d40b08e0 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa6cc001a10 con 0x7fa6d40b08e0 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6cbfff640 10 monclient: dump: 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.130 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6db78c640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.146+0000 7fa6db78c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6cc001b70 con 0x7fa6d40b08e0 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6cbfff640 1 -- 172.21.15.52:0/1557219223 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa6cc001f80 con 0x7fa6d40b08e0 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6cbfff640 10 monclient: dump: 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.131 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6db78c640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:26.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.147+0000 7fa6db78c640 4 set_mon_vals no callback set 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 10 monclient: shutdown 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 1 -- 172.21.15.52:0/1557219223 >> v1:172.21.15.79:6789/0 conn(0x7fa6d40b08e0 legacy=0x7fa6d413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 1 -- 172.21.15.52:0/1557219223 shutdown_connections 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 1 -- 172.21.15.52:0/1557219223 >> 172.21.15.52:0/1557219223 conn(0x7fa6d406b660 msgr2=0x7fa6d406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 1 -- 172.21.15.52:0/1557219223 shutdown_connections 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.148+0000 7fa6db78c640 1 -- 172.21.15.52:0/1557219223 wait complete. 2024-03-29T11:23:26.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.149+0000 7fa6db78c640 10 monclient: build_initial_monmap 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.149+0000 7fa6db78c640 10 monclient: monmap: 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.149+0000 7fa6db78c640 1 librados: starting msgr at 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.149+0000 7fa6db78c640 1 librados: starting objecter 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.149+0000 7fa6db78c640 1 Processor -- start 2024-03-29T11:23:26.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 1 -- start start 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 1 librados: setting wanted keys 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 1 librados: calling monclient init 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 10 monclient: init 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d40aeca0 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.150+0000 7fa6db78c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d40b0500 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6db78c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d413ba40 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6db78c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d41caa50 con 0x7fa6d40aeca0 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6db78c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d41cc080 con 0x7fa6d413ba40 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6db78c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d413c4e0 con 0x7fa6d40b0500 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6db78c640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6db78c640 10 monclient(hunting): authenticate will time out at 1136.291215s 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6d9528640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fa6d40aeca0 0x7fa6d41d54b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:51796/0 (socket says 172.21.15.52:51796) 2024-03-29T11:23:26.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6d9528640 1 -- 172.21.15.52:0/4080960047 learned_addr learned my addr 172.21.15.52:0/4080960047 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 client.?.objecter ms_handle_connect 0x7fa6d40b0500 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 client.?.objecter ms_handle_connect 0x7fa6d413ba40 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 client.?.objecter ms_handle_connect 0x7fa6d40aeca0 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921013784 0 0) 0x7fa6d41cc080 con 0x7fa6d413ba40 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8004eb0 con 0x7fa6d413ba40 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807179544 0 0) 0x7fa6d413c4e0 con 0x7fa6d40b0500 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.151+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d41cc080 con 0x7fa6d40b0500 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218188679 0 0) 0x7fa6d41caa50 con 0x7fa6d40aeca0 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d413c4e0 con 0x7fa6d40aeca0 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 802887273 0 0) 0x7fa6b8004eb0 con 0x7fa6d413ba40 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6d41caa50 con 0x7fa6d413ba40 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa6c4001870 con 0x7fa6d413ba40 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:26.140 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 68510376 0 0) 0x7fa6d41cc080 con 0x7fa6d40b0500 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b8004eb0 con 0x7fa6d40b0500 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa6cc001e80 con 0x7fa6d40b0500 2024-03-29T11:23:26.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2020600262 0 0) 0x7fa6d413c4e0 con 0x7fa6d40aeca0 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6d41cc080 con 0x7fa6d40aeca0 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa6d00018f0 con 0x7fa6d40aeca0 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.152+0000 7fa6c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:26.143 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2653902136 0 0) 0x7fa6d41caa50 con 0x7fa6d413ba40 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 >> v1:172.21.15.52:6790/0 conn(0x7fa6d40b0500 legacy=0x7fa6d41caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 >> v1:172.21.15.52:6789/0 conn(0x7fa6d40aeca0 legacy=0x7fa6d41d54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 1 monclient: found mon.b 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d413c670 con 0x7fa6d413ba40 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 10 monclient: _renew_subs 2024-03-29T11:23:26.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d41e0f00 con 0x7fa6d413ba40 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 10 monclient: _renew_subs 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d41e1270 con 0x7fa6d413ba40 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6db78c640 1 librados: init done 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6c4001d00 con 0x7fa6d413ba40 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.153+0000 7fa6cb7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.154+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa6c40037b0 con 0x7fa6d413ba40 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.154+0000 7fa6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.154+0000 7fa6c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.154+0000 7fa6c9ffb640 10 monclient: dump: 2024-03-29T11:23:26.145 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.155+0000 7fa6db78c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.155+0000 7fa6db78c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:26.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.155+0000 7fa6db78c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.155+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b4005080 con 0x7fa6d413ba40 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.155+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fa6c4019780 con 0x7fa6d413ba40 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.156+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fa6c4083780 con 0x7fa6d413ba40 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.156+0000 7fa6c9ffb640 10 client.4345.objecter ms_dispatch 0x7fa6d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.156+0000 7fa6c9ffb640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.156+0000 7fa6c9ffb640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.156+0000 7fa6c9ffb640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.156+0000 7fa6d9528640 10 monclient: get_auth_request con 0x7fa6b806e210 auth_method 0 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.157+0000 7fa6c9ffb640 10 client.4345.objecter ms_handle_connect 0x7fa6b806e210 2024-03-29T11:23:26.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.164+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6c404ee90 con 0x7fa6d413ba40 2024-03-29T11:23:26.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.164+0000 7fa6c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:26.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.164+0000 7fa6c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:26.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.556+0000 7fa6db78c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:26.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.556+0000 7fa6db78c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:26.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.556+0000 7fa6db78c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:23:26.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.556+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6b4005330 con 0x7fa6d413ba40 2024-03-29T11:23:26.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.557+0000 7fa6c9ffb640 1 -- 172.21.15.52:0/4080960047 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fa6c4050f90 con 0x7fa6d413ba40 2024-03-29T11:23:26.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.557+0000 7fa6c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:26.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.557+0000 7fa6c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:26.538 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:26.538 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 10 librados: watch_flush enter 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 10 librados: watch_flush exit 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 10 client.4345.objecter successfully canceled tick 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 >> v1:172.21.15.79:6816/407480584 conn(0x7fa6b806e210 legacy=0x7fa6b80706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 10 monclient: shutdown 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.561+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 >> v1:172.21.15.79:6789/0 conn(0x7fa6d413ba40 legacy=0x7fa6d41cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.562+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 shutdown_connections 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.562+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 >> 172.21.15.52:0/4080960047 conn(0x7fa6d406b660 msgr2=0x7fa6d40a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.562+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 shutdown_connections 2024-03-29T11:23:26.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.562+0000 7fa6db78c640 1 -- 172.21.15.52:0/4080960047 wait complete. 2024-03-29T11:23:26.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:26.562+0000 7fa6db78c640 1 librados: shutdown 2024-03-29T11:23:26.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:33.363 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:33.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.516+0000 7fade223e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.516+0000 7fade223e640 10 monclient: build_initial_monmap 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.516+0000 7fade223e640 10 monclient: monmap: 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:33.517383+0000 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:33.517383+0000 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:33.496 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:33.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.518+0000 7fade223e640 1 Processor -- start 2024-03-29T11:23:33.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.518+0000 7fade223e640 1 -- start start 2024-03-29T11:23:33.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.518+0000 7fade223e640 10 monclient: init 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient(hunting): picked mon.noname-a con 0x7faddc0b1f60 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient(hunting): picked mon.noname-b con 0x7faddc071eb0 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient(hunting): picked mon.noname-c con 0x7faddc143d10 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc06ef80 con 0x7faddc0b1f60 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc06f260 con 0x7faddc143d10 2024-03-29T11:23:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0b40c0 con 0x7faddc071eb0 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fade223e640 10 monclient(hunting): authenticate will time out at 1143.659699s 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7faddaffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7faddc071eb0 0x7faddc146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:46532/0 (socket says 172.21.15.52:46532) 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7faddaffd640 1 -- 172.21.15.52:0/2574014319 learned_addr learned my addr 172.21.15.52:0/2574014319 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783415811 0 0) 0x7faddc0b40c0 con 0x7faddc071eb0 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fadda7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fadda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fadda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fadda7fc640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.519+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4003520 con 0x7faddc071eb0 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.520+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200609773 0 0) 0x7fadc4003520 con 0x7faddc071eb0 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.520+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 >> v1:172.21.15.79:6789/0 conn(0x7faddc143d10 legacy=0x7faddc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.520+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 >> v1:172.21.15.52:6789/0 conn(0x7faddc0b1f60 legacy=0x7faddc143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc155d70 con 0x7faddc071eb0 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:33.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fadd0001690 con 0x7faddc071eb0 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: dump: 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:33.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd0001860 con 0x7faddc071eb0 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 1 -- 172.21.15.52:0/2574014319 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fadd0001cb0 con 0x7faddc071eb0 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.522+0000 7fadda7fc640 10 monclient: dump: 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:33.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.523+0000 7fade223e640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:23:33.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.523+0000 7fade223e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:33.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.523+0000 7fade223e640 4 set_mon_vals no callback set 2024-03-29T11:23:33.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 10 monclient: shutdown 2024-03-29T11:23:33.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:33.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 1 -- 172.21.15.52:0/2574014319 >> v1:172.21.15.52:6790/0 conn(0x7faddc071eb0 legacy=0x7faddc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 1 -- 172.21.15.52:0/2574014319 shutdown_connections 2024-03-29T11:23:33.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 1 -- 172.21.15.52:0/2574014319 >> 172.21.15.52:0/2574014319 conn(0x7faddc066a00 msgr2=0x7faddc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:33.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 1 -- 172.21.15.52:0/2574014319 shutdown_connections 2024-03-29T11:23:33.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.526+0000 7fade223e640 1 -- 172.21.15.52:0/2574014319 wait complete. 2024-03-29T11:23:33.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.528+0000 7fade223e640 10 monclient: build_initial_monmap 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.528+0000 7fade223e640 10 monclient: monmap: 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.528+0000 7fade223e640 1 librados: starting msgr at 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.528+0000 7fade223e640 1 librados: starting objecter 2024-03-29T11:23:33.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.529+0000 7fade223e640 1 Processor -- start 2024-03-29T11:23:33.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.529+0000 7fade223e640 1 -- start start 2024-03-29T11:23:33.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.529+0000 7fade223e640 1 librados: setting wanted keys 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.529+0000 7fade223e640 1 librados: calling monclient init 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.529+0000 7fade223e640 10 monclient: init 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 10 monclient(hunting): picked mon.noname-b con 0x7faddc0b1f60 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 10 monclient(hunting): picked mon.noname-a con 0x7faddc143d10 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 10 monclient(hunting): picked mon.noname-c con 0x7faddc140960 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc070bd0 con 0x7faddc143d10 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc155540 con 0x7faddc0b1f60 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7fade223e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc1dfa70 con 0x7faddc140960 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.530+0000 7faddb7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7faddc0b1f60 0x7faddc06f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:39916/0 (socket says 172.21.15.52:39916) 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7faddb7fe640 1 -- 172.21.15.52:0/1236538422 learned_addr learned my addr 172.21.15.52:0/1236538422 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fade223e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:33.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fade223e640 10 monclient(hunting): authenticate will time out at 1143.671217s 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 client.?.objecter ms_handle_connect 0x7faddc140960 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 client.?.objecter ms_handle_connect 0x7faddc0b1f60 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113432422 0 0) 0x7faddc1dfa70 con 0x7faddc140960 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc8004eb0 con 0x7faddc140960 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260600793 0 0) 0x7faddc155540 con 0x7faddc0b1f60 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.531+0000 7fadc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faddc1dfa70 con 0x7faddc0b1f60 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209259173 0 0) 0x7fadc8004eb0 con 0x7faddc140960 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faddc155540 con 0x7faddc140960 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fadcc001750 con 0x7faddc140960 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): dump: 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:33.515 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:33.516 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:33.516 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:33.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:33.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.532+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 336468211 0 0) 0x7faddc155540 con 0x7faddc140960 2024-03-29T11:23:33.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.533+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 >> v1:172.21.15.79:6789/0 conn(0x7faddc0b1f60 legacy=0x7faddc06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.533+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 >> v1:172.21.15.52:6789/0 conn(0x7faddc143d10 legacy=0x7faddc1dec50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:33.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:33.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 1 monclient: found mon.c 2024-03-29T11:23:33.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc070e00 con 0x7faddc140960 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fade223e640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadcc001c60 con 0x7faddc140960 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fade223e640 10 monclient: _renew_subs 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fade223e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc06fd30 con 0x7faddc140960 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fadcc003790 con 0x7faddc140960 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fade223e640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.534+0000 7fadc3fff640 10 monclient: dump: 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.535+0000 7fadd97fa640 4 set_mon_vals no callback set 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.536+0000 7fade223e640 10 monclient: _renew_subs 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.536+0000 7fade223e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.536+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc154990 con 0x7faddc140960 2024-03-29T11:23:33.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.536+0000 7fade223e640 1 librados: init done 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.537+0000 7fade223e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.537+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fadcc0038f0 con 0x7faddc140960 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.537+0000 7fade223e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.537+0000 7fade223e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.537+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc075f10 con 0x7faddc140960 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fadcc0847f0 con 0x7faddc140960 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7fadc3fff640 10 client.4364.objecter ms_dispatch 0x7faddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7fadc3fff640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7faddb7fe640 10 monclient: get_auth_request con 0x7fadc806da30 auth_method 0 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7fadc3fff640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7fadc3fff640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:23:33.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.538+0000 7fadc3fff640 10 client.4364.objecter ms_handle_connect 0x7fadc806da30 2024-03-29T11:23:33.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.546+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fadcc0b28b0 con 0x7faddc140960 2024-03-29T11:23:33.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.546+0000 7fadc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:33.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.546+0000 7fadc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:33.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.866+0000 7fade223e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.866+0000 7fade223e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.866+0000 7fade223e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:33.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.866+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faddc074f50 con 0x7faddc140960 2024-03-29T11:23:33.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.867+0000 7fadc3fff640 1 -- 172.21.15.52:0/1236538422 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fadcc04f0a0 con 0x7faddc140960 2024-03-29T11:23:33.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.867+0000 7fadc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.867+0000 7fadc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:33.847 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:33.847 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:33.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 10 librados: watch_flush enter 2024-03-29T11:23:33.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 10 librados: watch_flush exit 2024-03-29T11:23:33.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:23:33.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 >> v1:172.21.15.79:6816/407480584 conn(0x7fadc806da30 legacy=0x7fadc806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 10 monclient: shutdown 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.870+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 >> v1:172.21.15.52:6790/0 conn(0x7faddc140960 legacy=0x7faddc1df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.871+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 shutdown_connections 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.871+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 >> 172.21.15.52:0/1236538422 conn(0x7faddc066a00 msgr2=0x7faddc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.871+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 shutdown_connections 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.871+0000 7fade223e640 1 -- 172.21.15.52:0/1236538422 wait complete. 2024-03-29T11:23:33.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:33.871+0000 7fade223e640 1 librados: shutdown 2024-03-29T11:23:33.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:40.674 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.844+0000 7fe6236d6640 10 monclient: get_monmap_and_config 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.844+0000 7fe6236d6640 10 monclient: build_initial_monmap 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.844+0000 7fe6236d6640 10 monclient: monmap: 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:40.845570+0000 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:40.845570+0000 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:40.824 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.845+0000 7fe6236d6640 1 Processor -- start 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.845+0000 7fe6236d6640 1 -- start start 2024-03-29T11:23:40.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.845+0000 7fe6236d6640 10 monclient: init 2024-03-29T11:23:40.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.846+0000 7fe6236d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:40.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.846+0000 7fe6236d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe61c1457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe61c13f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe61c0a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c0aaef0 con 0x7fe61c1457f0 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c145bd0 con 0x7fe61c0a9d70 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c145eb0 con 0x7fe61c13f650 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.847+0000 7fe6236d6640 10 monclient(hunting): authenticate will time out at 1150.987922s 2024-03-29T11:23:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.850+0000 7fe621472640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe61c1457f0 0x7fe61c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:36529/0 (socket says 172.21.15.52:36529) 2024-03-29T11:23:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.850+0000 7fe621472640 1 -- 172.21.15.52:0/3842096138 learned_addr learned my addr 172.21.15.52:0/3842096138 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:40.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311099672 0 0) 0x7fe61c145eb0 con 0x7fe61c13f650 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe608003520 con 0x7fe61c13f650 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414001307 0 0) 0x7fe608003520 con 0x7fe61c13f650 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 >> v1:172.21.15.79:6789/0 conn(0x7fe61c0a9d70 legacy=0x7fe61c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 >> v1:172.21.15.52:6789/0 conn(0x7fe61c1457f0 legacy=0x7fe61c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 monclient: found mon.noname-b 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:40.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c146340 con 0x7fe61c13f650 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe60c001690 con 0x7fe61c13f650 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.851+0000 7fe613fff640 10 monclient: dump: 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.833 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe60c0017f0 con 0x7fe61c13f650 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe613fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe613fff640 1 -- 172.21.15.52:0/3842096138 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe60c001c40 con 0x7fe61c13f650 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe613fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe613fff640 10 monclient: dump: 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:40.834 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:40.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 4 set_mon_vals no callback set 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 10 monclient: shutdown 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.852+0000 7fe6236d6640 1 -- 172.21.15.52:0/3842096138 >> v1:172.21.15.52:6790/0 conn(0x7fe61c13f650 legacy=0x7fe61c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.853+0000 7fe6236d6640 1 -- 172.21.15.52:0/3842096138 shutdown_connections 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.853+0000 7fe6236d6640 1 -- 172.21.15.52:0/3842096138 >> 172.21.15.52:0/3842096138 conn(0x7fe61c066a00 msgr2=0x7fe61c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.853+0000 7fe6236d6640 1 -- 172.21.15.52:0/3842096138 shutdown_connections 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.853+0000 7fe6236d6640 1 -- 172.21.15.52:0/3842096138 wait complete. 2024-03-29T11:23:40.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 10 monclient: build_initial_monmap 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 10 monclient: monmap: 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 1 librados: starting msgr at 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 1 librados: starting objecter 2024-03-29T11:23:40.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 1 Processor -- start 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 1 -- start start 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.854+0000 7fe6236d6640 1 librados: setting wanted keys 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.855+0000 7fe6236d6640 1 librados: calling monclient init 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.855+0000 7fe6236d6640 10 monclient: init 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe61c0a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe61c13f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe61c1457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c06eab0 con 0x7fe61c0a9d70 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c141b20 con 0x7fe61c13f650 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c13fda0 con 0x7fe61c1457f0 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:40.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.856+0000 7fe6236d6640 10 monclient(hunting): authenticate will time out at 1150.997038s 2024-03-29T11:23:40.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.860+0000 7fe620c71640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fe61c13f650 0x7fe61c142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:43112/0 (socket says 172.21.15.52:43112) 2024-03-29T11:23:40.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.860+0000 7fe620c71640 1 -- 172.21.15.52:0/4125353944 learned_addr learned my addr 172.21.15.52:0/4125353944 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:40.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.860+0000 7fe611ffb640 10 client.?.objecter ms_handle_connect 0x7fe61c1457f0 2024-03-29T11:23:40.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 client.?.objecter ms_handle_connect 0x7fe61c13f650 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603273292 0 0) 0x7fe61c13fda0 con 0x7fe61c1457f0 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe600004eb0 con 0x7fe61c1457f0 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 client.?.objecter ms_handle_connect 0x7fe61c0a9d70 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210147636 0 0) 0x7fe61c141b20 con 0x7fe61c13f650 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61c13fda0 con 0x7fe61c13f650 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2974134637 0 0) 0x7fe600004eb0 con 0x7fe61c1457f0 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe61c141b20 con 0x7fe61c1457f0 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638067271 0 0) 0x7fe61c06eab0 con 0x7fe61c0a9d70 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe600004eb0 con 0x7fe61c0a9d70 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe614001bc0 con 0x7fe61c1457f0 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:40.848 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4020814984 0 0) 0x7fe61c141b20 con 0x7fe61c1457f0 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 >> v1:172.21.15.79:6789/0 conn(0x7fe61c13f650 legacy=0x7fe61c142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 >> v1:172.21.15.52:6789/0 conn(0x7fe61c0a9d70 legacy=0x7fe61c142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:40.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 1 monclient: found mon.c 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.861+0000 7fe611ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe611ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c139540 con 0x7fe61c1457f0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe611ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe611ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 10 monclient: _renew_subs 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61c1397f0 con 0x7fe61c1457f0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 10 monclient: _renew_subs 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c141630 con 0x7fe61c1457f0 2024-03-29T11:23:40.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.862+0000 7fe6236d6640 1 librados: init done 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.863+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe614001d20 con 0x7fe61c1457f0 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.863+0000 7fe611ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.863+0000 7fe612ffd640 4 set_mon_vals no callback set 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe614003b50 con 0x7fe61c1457f0 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe611ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe6236d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe611ffb640 10 monclient: dump: 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.851 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fe614003d70 con 0x7fe61c1457f0 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe6236d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe6236d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.865+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc005080 con 0x7fe61c1457f0 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.866+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fe614083b40 con 0x7fe61c1457f0 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.866+0000 7fe611ffb640 10 client.4370.objecter ms_dispatch 0x7fe61c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.866+0000 7fe621472640 10 monclient: get_auth_request con 0x7fe60006da80 auth_method 0 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.867+0000 7fe611ffb640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.867+0000 7fe611ffb640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.867+0000 7fe611ffb640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:23:40.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.867+0000 7fe611ffb640 10 client.4370.objecter ms_handle_connect 0x7fe60006da80 2024-03-29T11:23:40.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.875+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe61404f500 con 0x7fe61c1457f0 2024-03-29T11:23:40.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.875+0000 7fe611ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:40.875+0000 7fe611ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.189+0000 7fe6236d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.189+0000 7fe6236d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.189+0000 7fe6236d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.189+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5fc005330 con 0x7fe61c1457f0 2024-03-29T11:23:41.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.190+0000 7fe611ffb640 1 -- 172.21.15.52:0/4125353944 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fe614051430 con 0x7fe61c1457f0 2024-03-29T11:23:41.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.190+0000 7fe611ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:41.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.190+0000 7fe611ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:41.171 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:41.171 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 10 librados: watch_flush enter 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 10 librados: watch_flush exit 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 >> v1:172.21.15.79:6816/407480584 conn(0x7fe60006da80 legacy=0x7fe60006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 10 monclient: shutdown 2024-03-29T11:23:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.194+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 >> v1:172.21.15.52:6790/0 conn(0x7fe61c1457f0 legacy=0x7fe61c1ddaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.195+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 shutdown_connections 2024-03-29T11:23:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.195+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 >> 172.21.15.52:0/4125353944 conn(0x7fe61c066a00 msgr2=0x7fe61c13cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.195+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 shutdown_connections 2024-03-29T11:23:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.195+0000 7fe6236d6640 1 -- 172.21.15.52:0/4125353944 wait complete. 2024-03-29T11:23:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:41.195+0000 7fe6236d6640 1 librados: shutdown 2024-03-29T11:23:41.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:47.997 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.173+0000 7fdb5256b640 10 monclient: get_monmap_and_config 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.173+0000 7fdb5256b640 10 monclient: build_initial_monmap 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.173+0000 7fdb5256b640 10 monclient: monmap: 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:48.175073+0000 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:48.175073+0000 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:48.154 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:48.155 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:48.155 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:48.155 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:48.155 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.175+0000 7fdb5256b640 1 Processor -- start 2024-03-29T11:23:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 1 -- start start 2024-03-29T11:23:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 10 monclient: init 2024-03-29T11:23:48.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:48.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:48.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4c0aad20 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 10 monclient(hunting): picked mon.noname-a con 0x7fdb4c06d8c0 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 10 monclient(hunting): picked mon.noname-c con 0x7fdb4c0afc40 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0af4d0 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0af7b0 con 0x7fdb4c0afc40 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.176+0000 7fdb5256b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0acab0 con 0x7fdb4c0aad20 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.177+0000 7fdb5256b640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:48.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.177+0000 7fdb5256b640 10 monclient(hunting): authenticate will time out at 1158.318072s 2024-03-29T11:23:48.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.180+0000 7fdb51569640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fdb4c0aad20 0x7fdb4c0a96c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43085/0 (socket says 172.21.15.52:43085) 2024-03-29T11:23:48.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.180+0000 7fdb51569640 1 -- 172.21.15.52:0/1234040369 learned_addr learned my addr 172.21.15.52:0/1234040369 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.182+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585102045 0 0) 0x7fdb4c0af4d0 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.182+0000 7fdb43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.182+0000 7fdb43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.182+0000 7fdb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.182+0000 7fdb43fff640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.182+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb38003520 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922798778 0 0) 0x7fdb38003520 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 >> v1:172.21.15.52:6790/0 conn(0x7fdb4c0aad20 legacy=0x7fdb4c0a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 >> v1:172.21.15.79:6789/0 conn(0x7fdb4c0afc40 legacy=0x7fdb4c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 monclient: found mon.noname-a 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:48.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c0aba30 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdb3c001690 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: dump: 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.166 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb3c0017f0 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 1 -- 172.21.15.52:0/1234040369 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdb3c001c40 con 0x7fdb4c06d8c0 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb43fff640 10 monclient: dump: 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.167 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.183+0000 7fdb5256b640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:48.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 4 set_mon_vals no callback set 2024-03-29T11:23:48.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 10 monclient: shutdown 2024-03-29T11:23:48.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:48.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 1 -- 172.21.15.52:0/1234040369 >> v1:172.21.15.52:6789/0 conn(0x7fdb4c06d8c0 legacy=0x7fdb4c0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 1 -- 172.21.15.52:0/1234040369 shutdown_connections 2024-03-29T11:23:48.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.184+0000 7fdb5256b640 1 -- 172.21.15.52:0/1234040369 >> 172.21.15.52:0/1234040369 conn(0x7fdb4c0673e0 msgr2=0x7fdb4c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:48.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.186+0000 7fdb5256b640 1 -- 172.21.15.52:0/1234040369 shutdown_connections 2024-03-29T11:23:48.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.188+0000 7fdb5256b640 1 -- 172.21.15.52:0/1234040369 wait complete. 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.189+0000 7fdb5256b640 10 monclient: build_initial_monmap 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.189+0000 7fdb5256b640 10 monclient: monmap: 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.189+0000 7fdb5256b640 1 librados: starting msgr at 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.189+0000 7fdb5256b640 1 librados: starting objecter 2024-03-29T11:23:48.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.189+0000 7fdb5256b640 1 Processor -- start 2024-03-29T11:23:48.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 1 -- start start 2024-03-29T11:23:48.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 1 librados: setting wanted keys 2024-03-29T11:23:48.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 1 librados: calling monclient init 2024-03-29T11:23:48.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 10 monclient: init 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 10 monclient(hunting): picked mon.noname-c con 0x7fdb4c0aad20 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.190+0000 7fdb5256b640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4c0afc40 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.191+0000 7fdb5256b640 10 monclient(hunting): picked mon.noname-a con 0x7fdb4c139700 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.191+0000 7fdb5256b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0b3350 con 0x7fdb4c139700 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.191+0000 7fdb5256b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c06faf0 con 0x7fdb4c0afc40 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.191+0000 7fdb5256b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c06fdd0 con 0x7fdb4c0aad20 2024-03-29T11:23:48.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.191+0000 7fdb5256b640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:48.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.191+0000 7fdb5256b640 10 monclient(hunting): authenticate will time out at 1158.332414s 2024-03-29T11:23:48.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.194+0000 7fdb51569640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fdb4c0aad20 0x7fdb4c06de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43201/0 (socket says 172.21.15.52:43201) 2024-03-29T11:23:48.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.194+0000 7fdb51569640 1 -- 172.21.15.52:0/830612957 learned_addr learned my addr 172.21.15.52:0/830612957 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:48.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.194+0000 7fdb41ffb640 10 client.?.objecter ms_handle_connect 0x7fdb4c0aad20 2024-03-29T11:23:48.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.194+0000 7fdb41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861575782 0 0) 0x7fdb4c06fdd0 con 0x7fdb4c0aad20 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30004eb0 con 0x7fdb4c0aad20 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 client.?.objecter ms_handle_connect 0x7fdb4c0afc40 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 client.?.objecter ms_handle_connect 0x7fdb4c139700 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112396301 0 0) 0x7fdb4c06faf0 con 0x7fdb4c0afc40 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:23:48.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb300051d0 con 0x7fdb4c0afc40 2024-03-29T11:23:48.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569103053 0 0) 0x7fdb4c0b3350 con 0x7fdb4c139700 2024-03-29T11:23:48.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.197+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c06faf0 con 0x7fdb4c139700 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2922603135 0 0) 0x7fdb4c06faf0 con 0x7fdb4c139700 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb300057d0 con 0x7fdb4c139700 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdb48001650 con 0x7fdb4c139700 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.182 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 186029878 0 0) 0x7fdb300057d0 con 0x7fdb4c139700 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 >> v1:172.21.15.52:6790/0 conn(0x7fdb4c0aad20 legacy=0x7fdb4c06de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 >> v1:172.21.15.79:6789/0 conn(0x7fdb4c0afc40 legacy=0x7fdb4c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 monclient: found mon.a 2024-03-29T11:23:48.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c142d20 con 0x7fdb4c139700 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb5256b640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb5256b640 10 monclient: _renew_subs 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.198+0000 7fdb5256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c142ff0 con 0x7fdb4c139700 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb5256b640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb5256b640 10 monclient: _renew_subs 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb5256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c1dee30 con 0x7fdb4c139700 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb5256b640 1 librados: init done 2024-03-29T11:23:48.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb48001bc0 con 0x7fdb4c139700 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.199+0000 7fdb437fe640 4 set_mon_vals no callback set 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.200+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fdb48003790 con 0x7fdb4c139700 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.200+0000 7fdb41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.200+0000 7fdb41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.200+0000 7fdb41ffb640 10 monclient: dump: 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:48.185 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:48.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.202+0000 7fdb5256b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:48.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.202+0000 7fdb5256b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.202+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fdb48019760 con 0x7fdb4c139700 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.202+0000 7fdb5256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.202+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c005080 con 0x7fdb4c139700 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.203+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fdb48083310 con 0x7fdb4c139700 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.203+0000 7fdb41ffb640 10 client.4371.objecter ms_dispatch 0x7fdb4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.203+0000 7fdb41ffb640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.203+0000 7fdb41ffb640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.203+0000 7fdb41ffb640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-29T11:23:48.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.203+0000 7fdb51569640 10 monclient: get_auth_request con 0x7fdb3006e490 auth_method 0 2024-03-29T11:23:48.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.206+0000 7fdb41ffb640 10 client.4371.objecter ms_handle_connect 0x7fdb3006e490 2024-03-29T11:23:48.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.211+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb4804ebf0 con 0x7fdb4c139700 2024-03-29T11:23:48.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.211+0000 7fdb41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:48.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.211+0000 7fdb41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.546+0000 7fdb5256b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.546+0000 7fdb5256b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.546+0000 7fdb5256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:23:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.546+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb2c005330 con 0x7fdb4c139700 2024-03-29T11:23:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.548+0000 7fdb41ffb640 1 -- 172.21.15.52:0/830612957 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fdb4804ebf0 con 0x7fdb4c139700 2024-03-29T11:23:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.548+0000 7fdb41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.548+0000 7fdb41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:48.531 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:48.531 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:48.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.551+0000 7fdb5256b640 10 librados: watch_flush enter 2024-03-29T11:23:48.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.551+0000 7fdb5256b640 10 librados: watch_flush exit 2024-03-29T11:23:48.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.551+0000 7fdb5256b640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.551+0000 7fdb5256b640 10 client.4371.objecter successfully canceled tick 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 >> v1:172.21.15.79:6816/407480584 conn(0x7fdb3006e490 legacy=0x7fdb30070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 10 monclient: shutdown 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 >> v1:172.21.15.52:6789/0 conn(0x7fdb4c139700 legacy=0x7fdb4c1dd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 shutdown_connections 2024-03-29T11:23:48.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 >> 172.21.15.52:0/830612957 conn(0x7fdb4c0673e0 msgr2=0x7fdb4c0ac3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:48.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 shutdown_connections 2024-03-29T11:23:48.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.552+0000 7fdb5256b640 1 -- 172.21.15.52:0/830612957 wait complete. 2024-03-29T11:23:48.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:48.553+0000 7fdb5256b640 1 librados: shutdown 2024-03-29T11:23:48.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:55.357 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.528+0000 7fc9acc84640 10 monclient: get_monmap_and_config 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.528+0000 7fc9acc84640 10 monclient: build_initial_monmap 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.528+0000 7fc9acc84640 10 monclient: monmap: 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:23:55.529868+0000 2024-03-29T11:23:55.510 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:23:55.529868+0000 2024-03-29T11:23:55.511 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:55.511 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:55.511 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:55.511 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:23:55.511 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:23:55.511 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:55.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.533+0000 7fc9acc84640 1 Processor -- start 2024-03-29T11:23:55.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.534+0000 7fc9acc84640 1 -- start start 2024-03-29T11:23:55.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.534+0000 7fc9acc84640 10 monclient: init 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.534+0000 7fc9acc84640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a80acfa0 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a8072f10 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a80ad380 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80aa9f0 con 0x7fc9a8072f10 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80ac4c0 con 0x7fc9a80ad380 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80ac7a0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:55.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.535+0000 7fc9acc84640 10 monclient(hunting): authenticate will time out at 1165.675163s 2024-03-29T11:23:55.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.538+0000 7fc9a77fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc9a80acfa0 0x7fc9a8141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37431/0 (socket says 172.21.15.52:37431) 2024-03-29T11:23:55.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.538+0000 7fc9a77fe640 1 -- 172.21.15.52:0/881340081 learned_addr learned my addr 172.21.15.52:0/881340081 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:55.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266150265 0 0) 0x7fc9a80ac4c0 con 0x7fc9a80ad380 2024-03-29T11:23:55.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc994003520 con 0x7fc9a80ad380 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061071890 0 0) 0x7fc9a80ac7a0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.540+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a80ac4c0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296677200 0 0) 0x7fc9a80ac4c0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 >> v1:172.21.15.79:6789/0 conn(0x7fc9a80ad380 legacy=0x7fc9a81497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 >> v1:172.21.15.52:6789/0 conn(0x7fc9a8072f10 legacy=0x7fc9a8141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:55.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80ab5a0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9acc84640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9acc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc9900015f0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: dump: 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:55.526 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9acc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 1 -- 172.21.15.52:0/881340081 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc990001810 con 0x7fc9a80acfa0 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9acc84640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:55.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.541+0000 7fc9acc84640 4 set_mon_vals no callback set 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.542+0000 7fc9acc84640 10 monclient: shutdown 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.542+0000 7fc9acc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.542+0000 7fc9acc84640 1 -- 172.21.15.52:0/881340081 >> v1:172.21.15.52:6790/0 conn(0x7fc9a80acfa0 legacy=0x7fc9a8141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.542+0000 7fc9acc84640 1 -- 172.21.15.52:0/881340081 shutdown_connections 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.542+0000 7fc9acc84640 1 -- 172.21.15.52:0/881340081 >> 172.21.15.52:0/881340081 conn(0x7fc9a80b3a30 msgr2=0x7fc9a80b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.543+0000 7fc9acc84640 1 -- 172.21.15.52:0/881340081 shutdown_connections 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.543+0000 7fc9acc84640 1 -- 172.21.15.52:0/881340081 wait complete. 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.544+0000 7fc9acc84640 10 monclient: build_initial_monmap 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.544+0000 7fc9acc84640 10 monclient: monmap: 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:55.528 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.544+0000 7fc9acc84640 1 librados: starting msgr at 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.544+0000 7fc9acc84640 1 librados: starting objecter 2024-03-29T11:23:55.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.544+0000 7fc9acc84640 1 Processor -- start 2024-03-29T11:23:55.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.545+0000 7fc9acc84640 1 -- start start 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.545+0000 7fc9acc84640 1 librados: setting wanted keys 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.545+0000 7fc9acc84640 1 librados: calling monclient init 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.545+0000 7fc9acc84640 10 monclient: init 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.545+0000 7fc9acc84640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.545+0000 7fc9acc84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a8072f10 addr v1:172.21.15.52:6789/0 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a80acfa0 addr v1:172.21.15.52:6790/0 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a80ad380 addr v1:172.21.15.79:6789/0 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a81de620 con 0x7fc9a8072f10 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a81336e0 con 0x7fc9a80ad380 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a8131b00 con 0x7fc9a80acfa0 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.546+0000 7fc9acc84640 10 monclient(hunting): authenticate will time out at 1165.686122s 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc9a6ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc9a80acfa0 0x7fc9a81313f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37513/0 (socket says 172.21.15.52:37513) 2024-03-29T11:23:55.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc9a6ffd640 1 -- 172.21.15.52:0/716765092 learned_addr learned my addr 172.21.15.52:0/716765092 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 10 client.?.objecter ms_handle_connect 0x7fc9a80acfa0 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784468747 0 0) 0x7fc9a8131b00 con 0x7fc9a80acfa0 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:23:55.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.549+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc988004eb0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518955425 0 0) 0x7fc988004eb0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a8131b00 con 0x7fc9a80acfa0 2024-03-29T11:23:55.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc9980019d0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:55.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 10 monclient(hunting): dump: 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 10 client.?.objecter ms_handle_connect 0x7fc9a8072f10 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.551+0000 7fc99ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.552+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939930205 0 0) 0x7fc9a8131b00 con 0x7fc9a80acfa0 2024-03-29T11:23:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.552+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 >> v1:172.21.15.79:6789/0 conn(0x7fc9a80ad380 legacy=0x7fc9a81deb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.552+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 >> v1:172.21.15.52:6789/0 conn(0x7fc9a8072f10 legacy=0x7fc9a8132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.552+0000 7fc99ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.552+0000 7fc99ffff640 1 monclient: found mon.c 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc99ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a8131c90 con 0x7fc9a80acfa0 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc99ffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc99ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc99ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 10 monclient: _renew_subs 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a8133de0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 10 monclient: _renew_subs 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81df3f0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.553+0000 7fc9acc84640 1 librados: init done 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.555+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc998001b30 con 0x7fc9a80acfa0 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.555+0000 7fc99ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:55.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.556+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc998003a60 con 0x7fc9a80acfa0 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.556+0000 7fc99ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.556+0000 7fc99ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.556+0000 7fc99ffff640 10 monclient: dump: 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:23:55.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.556+0000 7fc9a5ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.559+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fc998003bc0 con 0x7fc9a80acfa0 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.559+0000 7fc9acc84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.559+0000 7fc9acc84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.559+0000 7fc9acc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.559+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c005080 con 0x7fc9a80acfa0 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc9a77fe640 10 monclient: get_auth_request con 0x7fc98806dca0 auth_method 0 2024-03-29T11:23:55.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fc998084b30 con 0x7fc9a80acfa0 2024-03-29T11:23:55.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc99ffff640 10 client.4379.objecter ms_dispatch 0x7fc9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:55.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc99ffff640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:55.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc99ffff640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:55.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc99ffff640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:23:55.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.560+0000 7fc99ffff640 10 client.4379.objecter ms_handle_connect 0x7fc98806dca0 2024-03-29T11:23:55.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.571+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc998050410 con 0x7fc9a80acfa0 2024-03-29T11:23:55.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.571+0000 7fc99ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:55.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.571+0000 7fc99ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.911+0000 7fc9acc84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.911+0000 7fc9acc84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.911+0000 7fc9acc84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.911+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc98c005330 con 0x7fc9a80acfa0 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.912+0000 7fc99ffff640 1 -- 172.21.15.52:0/716765092 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fc998052340 con 0x7fc9a80acfa0 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.912+0000 7fc99ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.912+0000 7fc99ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:55.893 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:23:55.893 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:55.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 10 librados: watch_flush enter 2024-03-29T11:23:55.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 10 librados: watch_flush exit 2024-03-29T11:23:55.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:23:55.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:23:55.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 >> v1:172.21.15.79:6816/407480584 conn(0x7fc98806dca0 legacy=0x7fc988070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 10 monclient: shutdown 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.916+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 >> v1:172.21.15.52:6790/0 conn(0x7fc9a80acfa0 legacy=0x7fc9a81313f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.917+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 shutdown_connections 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.917+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 >> 172.21.15.52:0/716765092 conn(0x7fc9a80b3a30 msgr2=0x7fc9a813a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.918+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 shutdown_connections 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.918+0000 7fc9acc84640 1 -- 172.21.15.52:0/716765092 wait complete. 2024-03-29T11:23:55.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:23:55.918+0000 7fc9acc84640 1 librados: shutdown 2024-03-29T11:23:55.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:02.722 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:02.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.892+0000 7fc541638640 10 monclient: get_monmap_and_config 2024-03-29T11:24:02.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.892+0000 7fc541638640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.892+0000 7fc541638640 10 monclient: monmap: 2024-03-29T11:24:02.876 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:02.876 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:02.893452+0000 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:02.893452+0000 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:02.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.895+0000 7fc541638640 1 Processor -- start 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.896+0000 7fc541638640 1 -- start start 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.896+0000 7fc541638640 10 monclient: init 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.897+0000 7fc541638640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.897+0000 7fc541638640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.897+0000 7fc541638640 10 monclient(hunting): picked mon.noname-b con 0x7fc53c0acfa0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.897+0000 7fc541638640 10 monclient(hunting): picked mon.noname-a con 0x7fc53c072f10 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:02.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.897+0000 7fc541638640 10 monclient(hunting): picked mon.noname-c con 0x7fc53c0ad380 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:02.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.898+0000 7fc541638640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0aa9f0 con 0x7fc53c072f10 2024-03-29T11:24:02.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.898+0000 7fc541638640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0ac4c0 con 0x7fc53c0ad380 2024-03-29T11:24:02.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.898+0000 7fc541638640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0ac7a0 con 0x7fc53c0acfa0 2024-03-29T11:24:02.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.898+0000 7fc541638640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.898+0000 7fc541638640 10 monclient(hunting): authenticate will time out at 1173.038741s 2024-03-29T11:24:02.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.902+0000 7fc53bfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc53c0acfa0 0x7fc53c141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39123/0 (socket says 172.21.15.52:39123) 2024-03-29T11:24:02.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.902+0000 7fc53bfff640 1 -- 172.21.15.52:0/3738701017 learned_addr learned my addr 172.21.15.52:0/3738701017 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:02.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.904+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107121588 0 0) 0x7fc53c0ac4c0 con 0x7fc53c0ad380 2024-03-29T11:24:02.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.904+0000 7fc53affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.904+0000 7fc53affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.904+0000 7fc53affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.904+0000 7fc53affd640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:24:02.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.904+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc524003520 con 0x7fc53c0ad380 2024-03-29T11:24:02.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043729026 0 0) 0x7fc53c0ac7a0 con 0x7fc53c0acfa0 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c0ac4c0 con 0x7fc53c0acfa0 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 860754210 0 0) 0x7fc524003520 con 0x7fc53c0ad380 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 >> v1:172.21.15.52:6790/0 conn(0x7fc53c0acfa0 legacy=0x7fc53c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 >> v1:172.21.15.52:6789/0 conn(0x7fc53c072f10 legacy=0x7fc53c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 monclient: found mon.noname-c 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:24:02.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0ab5a0 con 0x7fc53c0ad380 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc541638640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc541638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc52c001690 con 0x7fc53c0ad380 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc53affd640 10 monclient: dump: 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.887 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.905+0000 7fc541638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.907+0000 7fc53affd640 1 -- 172.21.15.52:0/3738701017 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc52c0018b0 con 0x7fc53c0ad380 2024-03-29T11:24:02.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.907+0000 7fc53affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.907+0000 7fc541638640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.907+0000 7fc541638640 4 set_mon_vals no callback set 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.908+0000 7fc541638640 10 monclient: shutdown 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.908+0000 7fc541638640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.908+0000 7fc541638640 1 -- 172.21.15.52:0/3738701017 >> v1:172.21.15.79:6789/0 conn(0x7fc53c0ad380 legacy=0x7fc53c1497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.909+0000 7fc541638640 1 -- 172.21.15.52:0/3738701017 shutdown_connections 2024-03-29T11:24:02.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.909+0000 7fc541638640 1 -- 172.21.15.52:0/3738701017 >> 172.21.15.52:0/3738701017 conn(0x7fc53c0b3a30 msgr2=0x7fc53c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.909+0000 7fc541638640 1 -- 172.21.15.52:0/3738701017 shutdown_connections 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.910+0000 7fc541638640 1 -- 172.21.15.52:0/3738701017 wait complete. 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.911+0000 7fc541638640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.911+0000 7fc541638640 10 monclient: monmap: 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.892 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.911+0000 7fc541638640 1 librados: starting msgr at 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.911+0000 7fc541638640 1 librados: starting objecter 2024-03-29T11:24:02.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.912+0000 7fc541638640 1 Processor -- start 2024-03-29T11:24:02.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.912+0000 7fc541638640 1 -- start start 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.912+0000 7fc541638640 1 librados: setting wanted keys 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.912+0000 7fc541638640 1 librados: calling monclient init 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.912+0000 7fc541638640 10 monclient: init 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient(hunting): picked mon.noname-c con 0x7fc53c072f10 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient(hunting): picked mon.noname-a con 0x7fc53c0acfa0 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient(hunting): picked mon.noname-b con 0x7fc53c0ad380 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c06ecf0 con 0x7fc53c0acfa0 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c070410 con 0x7fc53c0ad380 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0706f0 con 0x7fc53c072f10 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.913+0000 7fc541638640 10 monclient(hunting): authenticate will time out at 1173.054046s 2024-03-29T11:24:02.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.917+0000 7fc53bfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc53c072f10 0x7fc53c06e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39243/0 (socket says 172.21.15.52:39243) 2024-03-29T11:24:02.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.917+0000 7fc540e37640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc53c0ad380 0x7fc53c13dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:58933/0 (socket says 172.21.15.52:58933) 2024-03-29T11:24:02.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.917+0000 7fc540e37640 1 -- 172.21.15.52:0/589199587 learned_addr learned my addr 172.21.15.52:0/589199587 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:02.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.917+0000 7fc538ff9640 10 client.?.objecter ms_handle_connect 0x7fc53c072f10 2024-03-29T11:24:02.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297079243 0 0) 0x7fc53c0706f0 con 0x7fc53c072f10 2024-03-29T11:24:02.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:24:02.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.918+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528004eb0 con 0x7fc53c072f10 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 10 client.?.objecter ms_handle_connect 0x7fc53c0ad380 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945816011 0 0) 0x7fc528004eb0 con 0x7fc53c072f10 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc528005300 con 0x7fc53c072f10 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc5300019f0 con 0x7fc53c072f10 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:02.901 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3262746130 0 0) 0x7fc528005300 con 0x7fc53c072f10 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 >> v1:172.21.15.79:6789/0 conn(0x7fc53c0ad380 legacy=0x7fc53c13dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.920+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 >> v1:172.21.15.52:6789/0 conn(0x7fc53c0acfa0 legacy=0x7fc53c13d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 1 monclient: found mon.c 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0709d0 con 0x7fc53c072f10 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 10 client.?.objecter ms_handle_connect 0x7fc53c0acfa0 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc538ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 10 monclient: _renew_subs 2024-03-29T11:24:02.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c06f670 con 0x7fc53c072f10 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 10 monclient: _renew_subs 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c1e0c60 con 0x7fc53c072f10 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.922+0000 7fc541638640 1 librados: init done 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5300047e0 con 0x7fc53c072f10 2024-03-29T11:24:02.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc538ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc530004bf0 con 0x7fc53c072f10 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc538ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc538ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc538ff9640 10 monclient: dump: 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:02.905 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:02.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.924+0000 7fc53a7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:02.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.925+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fc530004dd0 con 0x7fc53c072f10 2024-03-29T11:24:02.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.927+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fc530004a10 con 0x7fc53c072f10 2024-03-29T11:24:02.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.927+0000 7fc538ff9640 10 client.4385.objecter ms_dispatch 0x7fc53c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:02.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.927+0000 7fc538ff9640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.927+0000 7fc538ff9640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.927+0000 7fc538ff9640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.928+0000 7fc53b7fe640 10 monclient: get_auth_request con 0x7fc52806e0c0 auth_method 0 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.928+0000 7fc541638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.928+0000 7fc541638640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.928+0000 7fc541638640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:02.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.928+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c005080 con 0x7fc53c072f10 2024-03-29T11:24:02.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.930+0000 7fc538ff9640 10 client.4385.objecter ms_handle_connect 0x7fc52806e0c0 2024-03-29T11:24:02.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.940+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc530050980 con 0x7fc53c072f10 2024-03-29T11:24:02.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.940+0000 7fc538ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:02.940+0000 7fc538ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:03.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.270+0000 7fc541638640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:03.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.270+0000 7fc541638640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:03.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.270+0000 7fc541638640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:03.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.270+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc51c005330 con 0x7fc53c072f10 2024-03-29T11:24:03.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.272+0000 7fc538ff9640 1 -- 172.21.15.52:0/589199587 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fc530050980 con 0x7fc53c072f10 2024-03-29T11:24:03.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.272+0000 7fc538ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:03.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.272+0000 7fc538ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:03.256 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:03.256 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:03.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.278+0000 7fc541638640 10 librados: watch_flush enter 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 10 librados: watch_flush exit 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 >> v1:172.21.15.79:6816/407480584 conn(0x7fc52806e0c0 legacy=0x7fc528070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 10 monclient: shutdown 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 >> v1:172.21.15.52:6790/0 conn(0x7fc53c072f10 legacy=0x7fc53c06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 shutdown_connections 2024-03-29T11:24:03.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 >> 172.21.15.52:0/589199587 conn(0x7fc53c0b3a30 msgr2=0x7fc53c124600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:03.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 shutdown_connections 2024-03-29T11:24:03.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.279+0000 7fc541638640 1 -- 172.21.15.52:0/589199587 wait complete. 2024-03-29T11:24:03.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:03.280+0000 7fc541638640 1 librados: shutdown 2024-03-29T11:24:03.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:10.086 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.260+0000 7fca7c64e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.260+0000 7fca7c64e640 10 monclient: build_initial_monmap 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.260+0000 7fca7c64e640 10 monclient: monmap: 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:10.261743+0000 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:10.261743+0000 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:10.242 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:10.243 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:10.243 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:10.243 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:10.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.262+0000 7fca7c64e640 1 Processor -- start 2024-03-29T11:24:10.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.262+0000 7fca7c64e640 1 -- start start 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.262+0000 7fca7c64e640 10 monclient: init 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b0860 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient(hunting): picked mon.noname-a con 0x7fca7413ef30 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient(hunting): picked mon.noname-b con 0x7fca740af8e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b0c40 con 0x7fca7413ef30 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740aed10 con 0x7fca740b0860 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740afd50 con 0x7fca740af8e0 2024-03-29T11:24:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.263+0000 7fca7c64e640 10 monclient(hunting): authenticate will time out at 1180.403243s 2024-03-29T11:24:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.267+0000 7fca7abeb640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fca740af8e0 0x7fca74149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52307/0 (socket says 172.21.15.52:52307) 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.267+0000 7fca7abeb640 1 -- 172.21.15.52:0/2325566872 learned_addr learned my addr 172.21.15.52:0/2325566872 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940520747 0 0) 0x7fca740afd50 con 0x7fca740af8e0 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca60003520 con 0x7fca740af8e0 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.268+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415174375 0 0) 0x7fca60003520 con 0x7fca740af8e0 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 >> v1:172.21.15.79:6789/0 conn(0x7fca740b0860 legacy=0x7fca7413e820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:24:10.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 >> v1:172.21.15.52:6789/0 conn(0x7fca7413ef30 legacy=0x7fca740af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: _finish_hunting 0 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 1 monclient: found mon.noname-b 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740ab1b0 con 0x7fca740af8e0 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: _finish_auth 0 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: _check_auth_tickets 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:10.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca7c64e640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca7c64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fca6c001a10 con 0x7fca740af8e0 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca793e8640 10 monclient: dump: 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:10.252 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:10.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.269+0000 7fca7c64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:10.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.272+0000 7fca793e8640 1 -- 172.21.15.52:0/2325566872 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca6c001ca0 con 0x7fca740af8e0 2024-03-29T11:24:10.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.272+0000 7fca793e8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:10.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.272+0000 7fca7c64e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:10.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.272+0000 7fca7c64e640 4 set_mon_vals no callback set 2024-03-29T11:24:10.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 10 monclient: shutdown 2024-03-29T11:24:10.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 1 -- 172.21.15.52:0/2325566872 >> v1:172.21.15.52:6790/0 conn(0x7fca740af8e0 legacy=0x7fca74149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 1 -- 172.21.15.52:0/2325566872 shutdown_connections 2024-03-29T11:24:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 1 -- 172.21.15.52:0/2325566872 >> 172.21.15.52:0/2325566872 conn(0x7fca7406b660 msgr2=0x7fca7406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 1 -- 172.21.15.52:0/2325566872 shutdown_connections 2024-03-29T11:24:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.273+0000 7fca7c64e640 1 -- 172.21.15.52:0/2325566872 wait complete. 2024-03-29T11:24:10.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.274+0000 7fca7c64e640 10 monclient: build_initial_monmap 2024-03-29T11:24:10.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.274+0000 7fca7c64e640 10 monclient: monmap: 2024-03-29T11:24:10.256 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:10.256 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:10.256 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.274+0000 7fca7c64e640 1 librados: starting msgr at 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.274+0000 7fca7c64e640 1 librados: starting objecter 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.275+0000 7fca7c64e640 1 Processor -- start 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.275+0000 7fca7c64e640 1 -- start start 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.275+0000 7fca7c64e640 1 librados: setting wanted keys 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.275+0000 7fca7c64e640 1 librados: calling monclient init 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.275+0000 7fca7c64e640 10 monclient: init 2024-03-29T11:24:10.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.275+0000 7fca7c64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:10.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.276+0000 7fca7c64e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:10.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.276+0000 7fca7c64e640 10 monclient(hunting): picked mon.noname-b con 0x7fca740af8e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:10.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.279+0000 7fca7c64e640 10 monclient(hunting): picked mon.noname-c con 0x7fca740b0860 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.279+0000 7fca7c64e640 10 monclient(hunting): picked mon.noname-a con 0x7fca7413ef30 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.282+0000 7fca79be9640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fca740b0860 0x7fca74132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52395/0 (socket says 172.21.15.52:52395) 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.282+0000 7fca79be9640 1 -- 172.21.15.52:0/2726393196 learned_addr learned my addr 172.21.15.52:0/2726393196 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.282+0000 7fca6affd640 10 client.?.objecter ms_handle_connect 0x7fca740b0860 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.282+0000 7fca6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca7c64e640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7406dc70 con 0x7fca7413ef30 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca7c64e640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca74133150 con 0x7fca740af8e0 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca7c64e640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b4110 con 0x7fca740b0860 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca7c64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca7c64e640 10 monclient(hunting): authenticate will time out at 1180.423475s 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 client.?.objecter ms_handle_connect 0x7fca7413ef30 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 client.?.objecter ms_handle_connect 0x7fca740af8e0 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684096263 0 0) 0x7fca7406dc70 con 0x7fca7413ef30 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.283+0000 7fca6affd640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca58004eb0 con 0x7fca7413ef30 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3280144803 0 0) 0x7fca58004eb0 con 0x7fca7413ef30 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca58005300 con 0x7fca7413ef30 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fca6c004e30 con 0x7fca7413ef30 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:10.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 10 monclient(hunting): dump: 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.284+0000 7fca6affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643788427 0 0) 0x7fca740b4110 con 0x7fca740b0860 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:10.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca58005880 con 0x7fca740b0860 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2032301355 0 0) 0x7fca58005300 con 0x7fca7413ef30 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 >> v1:172.21.15.52:6790/0 conn(0x7fca740b0860 legacy=0x7fca74132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 >> v1:172.21.15.79:6789/0 conn(0x7fca740af8e0 legacy=0x7fca741df940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 monclient: found mon.a 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740b42a0 con 0x7fca7413ef30 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.285+0000 7fca6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:24:10.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 10 monclient: _renew_subs 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca740b3570 con 0x7fca7413ef30 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 10 monclient: _renew_subs 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca7c64e640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca741df2c0 con 0x7fca7413ef30 2024-03-29T11:24:10.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca6c005240 con 0x7fca7413ef30 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fca6c004e30 con 0x7fca7413ef30 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca6affd640 10 monclient: dump: 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:10.273 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:10.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.286+0000 7fca78be7640 4 set_mon_vals no callback set 2024-03-29T11:24:10.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.287+0000 7fca7c64e640 1 librados: init done 2024-03-29T11:24:10.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.292+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fca6c01b220 con 0x7fca7413ef30 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fca6c001e90 con 0x7fca7413ef30 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca6affd640 10 client.4383.objecter ms_dispatch 0x7fca74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca6affd640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca6affd640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca68ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca68ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca68ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca6affd640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.293+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50005080 con 0x7fca7413ef30 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.294+0000 7fca7a3ea640 10 monclient: get_auth_request con 0x7fca5806e350 auth_method 0 2024-03-29T11:24:10.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.294+0000 7fca6affd640 10 client.4383.objecter ms_handle_connect 0x7fca5806e350 2024-03-29T11:24:10.283 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.302+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca6c050a80 con 0x7fca7413ef30 2024-03-29T11:24:10.283 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.302+0000 7fca6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:10.283 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.302+0000 7fca6affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:10.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.629+0000 7fca68ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.629+0000 7fca68ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.629+0000 7fca68ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:10.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.629+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca50005330 con 0x7fca7413ef30 2024-03-29T11:24:10.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.637+0000 7fca6affd640 1 -- 172.21.15.52:0/2726393196 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fca6c0847f0 con 0x7fca7413ef30 2024-03-29T11:24:10.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.637+0000 7fca6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.637+0000 7fca6affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:10.618 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:10.619 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:10.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.644+0000 7fca68ff9640 10 librados: watch_flush enter 2024-03-29T11:24:10.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.644+0000 7fca68ff9640 10 librados: watch_flush exit 2024-03-29T11:24:10.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.644+0000 7fca68ff9640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-29T11:24:10.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.644+0000 7fca68ff9640 10 client.4383.objecter successfully canceled tick 2024-03-29T11:24:10.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.645+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 >> v1:172.21.15.79:6816/407480584 conn(0x7fca5806e350 legacy=0x7fca58070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.645+0000 7fca68ff9640 10 monclient: shutdown 2024-03-29T11:24:10.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.645+0000 7fca68ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:10.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.645+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 >> v1:172.21.15.52:6789/0 conn(0x7fca7413ef30 legacy=0x7fca74132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.645+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 shutdown_connections 2024-03-29T11:24:10.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.645+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 >> 172.21.15.52:0/2726393196 conn(0x7fca7406b660 msgr2=0x7fca7413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:10.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.646+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 shutdown_connections 2024-03-29T11:24:10.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.646+0000 7fca68ff9640 1 -- 172.21.15.52:0/2726393196 wait complete. 2024-03-29T11:24:10.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:10.646+0000 7fca68ff9640 1 librados: shutdown 2024-03-29T11:24:10.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:17.451 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.623+0000 7f1014b55640 10 monclient: get_monmap_and_config 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.623+0000 7f1014b55640 10 monclient: build_initial_monmap 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.623+0000 7f1014b55640 10 monclient: monmap: 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.604 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:17.624117+0000 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:17.624117+0000 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:17.605 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.624+0000 7f1014b55640 1 Processor -- start 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.624+0000 7f1014b55640 1 -- start start 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.624+0000 7f1014b55640 10 monclient: init 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.625+0000 7f1014b55640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.625+0000 7f1014b55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.625+0000 7f1014b55640 10 monclient(hunting): picked mon.noname-a con 0x7f10100b01a0 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:17.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.625+0000 7f1014b55640 10 monclient(hunting): picked mon.noname-b con 0x7f10100af010 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:17.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.625+0000 7f1014b55640 10 monclient(hunting): picked mon.noname-c con 0x7f10100ac5a0 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.626+0000 7f1014b55640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100ae8a0 con 0x7f10100b01a0 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.626+0000 7f1014b55640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100ab1d0 con 0x7f10100ac5a0 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.626+0000 7f1014b55640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100ab4b0 con 0x7f10100af010 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.626+0000 7f1014b55640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.626+0000 7f1014b55640 10 monclient(hunting): authenticate will time out at 1187.766882s 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.627+0000 7f100e59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f10100b01a0 0x7f10100abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:45644/0 (socket says 172.21.15.52:45644) 2024-03-29T11:24:17.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.627+0000 7f100e59c640 1 -- 172.21.15.52:0/1764391433 learned_addr learned my addr 172.21.15.52:0/1764391433 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:17.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329440055 0 0) 0x7f10100ae8a0 con 0x7f10100b01a0 2024-03-29T11:24:17.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:24:17.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff8003520 con 0x7f10100b01a0 2024-03-29T11:24:17.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621036616 0 0) 0x7f10100ab1d0 con 0x7f10100ac5a0 2024-03-29T11:24:17.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.629+0000 7f100d59a640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:24:17.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10100ae8a0 con 0x7f10100ac5a0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4185183392 0 0) 0x7f0ff8003520 con 0x7f10100b01a0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 >> v1:172.21.15.52:6790/0 conn(0x7f10100af010 legacy=0x7f1010149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 >> v1:172.21.15.79:6789/0 conn(0x7f10100ac5a0 legacy=0x7f1010151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 monclient: found mon.noname-a 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10100ab790 con 0x7f10100b01a0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: _finish_auth 0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1000001690 con 0x7f10100b01a0 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:17.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: dump: 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10000018b0 con 0x7f10100b01a0 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 1 -- 172.21.15.52:0/1764391433 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1000001c40 con 0x7f10100b01a0 2024-03-29T11:24:17.616 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f100d59a640 10 monclient: dump: 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:17.617 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f1014b55640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f1014b55640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.630+0000 7f1014b55640 4 set_mon_vals no callback set 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.631+0000 7f1014b55640 10 monclient: shutdown 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.631+0000 7f1014b55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.631+0000 7f1014b55640 1 -- 172.21.15.52:0/1764391433 >> v1:172.21.15.52:6789/0 conn(0x7f10100b01a0 legacy=0x7f10100abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.631+0000 7f1014b55640 1 -- 172.21.15.52:0/1764391433 shutdown_connections 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.631+0000 7f1014b55640 1 -- 172.21.15.52:0/1764391433 >> 172.21.15.52:0/1764391433 conn(0x7f101006b660 msgr2=0x7f101006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:17.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.633+0000 7f1014b55640 1 -- 172.21.15.52:0/1764391433 shutdown_connections 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.634+0000 7f1014b55640 1 -- 172.21.15.52:0/1764391433 wait complete. 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.635+0000 7f1014b55640 10 monclient: build_initial_monmap 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.635+0000 7f1014b55640 10 monclient: monmap: 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:17.619 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.635+0000 7f1014b55640 1 librados: starting msgr at 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.635+0000 7f1014b55640 1 librados: starting objecter 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.636+0000 7f1014b55640 1 Processor -- start 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 1 -- start start 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 1 librados: setting wanted keys 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 1 librados: calling monclient init 2024-03-29T11:24:17.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient: init 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient(hunting): picked mon.noname-c con 0x7f10100ac5a0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient(hunting): picked mon.noname-b con 0x7f10100b01a0 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient(hunting): picked mon.noname-a con 0x7f10101235f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10101e3d50 con 0x7f10101235f0 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1010122490 con 0x7f10100b01a0 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10101e4bd0 con 0x7f10100ac5a0 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:17.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.637+0000 7f1014b55640 10 monclient(hunting): authenticate will time out at 1187.777971s 2024-03-29T11:24:17.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.640+0000 7f100dd9b640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f10100b01a0 0x7f1010122ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:42043/0 (socket says 172.21.15.52:42043) 2024-03-29T11:24:17.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.640+0000 7f100dd9b640 1 -- 172.21.15.52:0/4125067264 learned_addr learned my addr 172.21.15.52:0/4125067264 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 client.?.objecter ms_handle_connect 0x7f10100b01a0 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 client.?.objecter ms_handle_connect 0x7f10101235f0 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 client.?.objecter ms_handle_connect 0x7f10100ac5a0 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346366071 0 0) 0x7f1010122490 con 0x7f10100b01a0 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.642+0000 7f0ffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff0004eb0 con 0x7f10100b01a0 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025347804 0 0) 0x7f10101e4bd0 con 0x7f10100ac5a0 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1010122490 con 0x7f10100ac5a0 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916686417 0 0) 0x7f10101e3d50 con 0x7f10101235f0 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.643+0000 7f0ffeffd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:24:17.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10101e4bd0 con 0x7f10101235f0 2024-03-29T11:24:17.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666765812 0 0) 0x7f0ff0004eb0 con 0x7f10100b01a0 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10101e3d50 con 0x7f10100b01a0 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755885800 0 0) 0x7f1010122490 con 0x7f10100ac5a0 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff0004eb0 con 0x7f10100ac5a0 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1008001ba0 con 0x7f10100b01a0 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 10 monclient(hunting): dump: 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.630 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.644+0000 7f0ffeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561881182 0 0) 0x7f10101e4bd0 con 0x7f10101235f0 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1010122490 con 0x7f10101235f0 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f10040018b0 con 0x7f10101235f0 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 10 monclient(hunting): dump: 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.631 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1000001a20 con 0x7f10100ac5a0 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 10 monclient(hunting): dump: 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.632 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2154233225 0 0) 0x7f1010122490 con 0x7f10101235f0 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 >> v1:172.21.15.52:6790/0 conn(0x7f10100ac5a0 legacy=0x7f10101e44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.645+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 >> v1:172.21.15.79:6789/0 conn(0x7f10100b01a0 legacy=0x7f1010122ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 1 monclient: found mon.a 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010122770 con 0x7f10101235f0 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:17.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 10 monclient: _renew_subs 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10101e4000 con 0x7f10101235f0 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 10 monclient: _renew_subs 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1010120410 con 0x7f10101235f0 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f1014b55640 1 librados: init done 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10040035f0 con 0x7f10101235f0 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffffff640 4 set_mon_vals no callback set 2024-03-29T11:24:17.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1004001d80 con 0x7f10101235f0 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.646+0000 7f0ffeffd640 10 monclient: dump: 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:17.635 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.647+0000 7f1014b55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.647+0000 7f1014b55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.647+0000 7f1014b55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.647+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff4005080 con 0x7f10101235f0 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.648+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f1004019770 con 0x7f10101235f0 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f100e59c640 10 monclient: get_auth_request con 0x7f0ff006e250 auth_method 0 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f10040836f0 con 0x7f10101235f0 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f0ffeffd640 10 client.4389.objecter ms_dispatch 0x7f1010000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f0ffeffd640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f0ffeffd640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f0ffeffd640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T11:24:17.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.649+0000 7f0ffeffd640 10 client.4389.objecter ms_handle_connect 0x7f0ff006e250 2024-03-29T11:24:17.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.658+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f100404efd0 con 0x7f10101235f0 2024-03-29T11:24:17.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.658+0000 7f0ffeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.658+0000 7f0ffeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:17.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.990+0000 7f1014b55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.990+0000 7f1014b55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.990+0000 7f1014b55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:17.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.990+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ff4005330 con 0x7f10101235f0 2024-03-29T11:24:17.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.991+0000 7f0ffeffd640 1 -- 172.21.15.52:0/4125067264 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f1004050f00 con 0x7f10101235f0 2024-03-29T11:24:17.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.991+0000 7f0ffeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.991+0000 7f0ffeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:17.973 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:17.973 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 10 librados: watch_flush enter 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 10 librados: watch_flush exit 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 10 client.4389.objecter successfully canceled tick 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 >> v1:172.21.15.79:6816/407480584 conn(0x7f0ff006e250 legacy=0x7f0ff0070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 10 monclient: shutdown 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 >> v1:172.21.15.52:6789/0 conn(0x7f10101235f0 legacy=0x7f10101ee7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 shutdown_connections 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 >> 172.21.15.52:0/4125067264 conn(0x7f101006b660 msgr2=0x7f10100b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 shutdown_connections 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 -- 172.21.15.52:0/4125067264 wait complete. 2024-03-29T11:24:17.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:17.995+0000 7f1014b55640 1 librados: shutdown 2024-03-29T11:24:17.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:24.800 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:24.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.969+0000 7f9a06f0e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.969+0000 7f9a06f0e640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.969+0000 7f9a06f0e640 10 monclient: monmap: 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:24.970033+0000 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:24.970033+0000 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:24.951 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.970+0000 7f9a06f0e640 1 Processor -- start 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.970+0000 7f9a06f0e640 1 -- start start 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.970+0000 7f9a06f0e640 10 monclient: init 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9a001457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0013f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f9a000a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a000aaef0 con 0x7f9a000a9d70 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a00145bd0 con 0x7f9a0013f650 2024-03-29T11:24:24.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a00145eb0 con 0x7f9a001457f0 2024-03-29T11:24:24.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f9a06f0e640 10 monclient(hunting): authenticate will time out at 1195.111224s 2024-03-29T11:24:24.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f99fffff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f9a0013f650 0x7f9a000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:57250/0 (socket says 172.21.15.52:57250) 2024-03-29T11:24:24.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.971+0000 7f99fffff640 1 -- 172.21.15.52:0/1628969085 learned_addr learned my addr 172.21.15.52:0/1628969085 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:24.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908378958 0 0) 0x7f9a000aaef0 con 0x7f9a000a9d70 2024-03-29T11:24:24.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:24:24.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0003520 con 0x7f9a000a9d70 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754419162 0 0) 0x7f9a00145bd0 con 0x7f9a0013f650 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a000aaef0 con 0x7f9a0013f650 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802240516 0 0) 0x7f9a00145eb0 con 0x7f9a001457f0 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a00145bd0 con 0x7f9a001457f0 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2470213086 0 0) 0x7f99f0003520 con 0x7f9a000a9d70 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 >> v1:172.21.15.52:6790/0 conn(0x7f9a001457f0 legacy=0x7f9a0013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 >> v1:172.21.15.79:6789/0 conn(0x7f9a0013f650 legacy=0x7f9a000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a00146340 con 0x7f9a000a9d70 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f99f8001a10 con 0x7f9a000a9d70 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f99ff7fe640 10 monclient: dump: 2024-03-29T11:24:24.958 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f9a06f0e640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.972+0000 7f9a06f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99f8001bf0 con 0x7f9a000a9d70 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f99ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f9a06f0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:24.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f9a06f0e640 4 set_mon_vals no callback set 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f99ff7fe640 1 -- 172.21.15.52:0/1628969085 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f99f8002000 con 0x7f9a000a9d70 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f99ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f99ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.973+0000 7f99ff7fe640 10 monclient: dump: 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:24.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.974+0000 7f9a06f0e640 10 monclient: shutdown 2024-03-29T11:24:24.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.974+0000 7f9a06f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.974+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1628969085 >> v1:172.21.15.52:6789/0 conn(0x7f9a000a9d70 legacy=0x7f9a0014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.974+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1628969085 shutdown_connections 2024-03-29T11:24:24.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.974+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1628969085 >> 172.21.15.52:0/1628969085 conn(0x7f9a00066a00 msgr2=0x7f9a00066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.974+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1628969085 shutdown_connections 2024-03-29T11:24:24.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.975+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1628969085 wait complete. 2024-03-29T11:24:24.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.976+0000 7f9a06f0e640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.976+0000 7f9a06f0e640 10 monclient: monmap: 2024-03-29T11:24:24.967 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:24.967 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:24.967 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 1 librados: starting msgr at 2024-03-29T11:24:24.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 1 librados: starting objecter 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 1 Processor -- start 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 1 -- start start 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 1 librados: setting wanted keys 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 1 librados: calling monclient init 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.977+0000 7f9a06f0e640 10 monclient: init 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9a000a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f9a0013f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f9a001457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a001cb8b0 con 0x7f9a0013f650 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a00138930 con 0x7f9a000a9d70 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a001cc7f0 con 0x7f9a001457f0 2024-03-29T11:24:24.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a06f0e640 10 monclient(hunting): authenticate will time out at 1195.119526s 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a04caa640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f9a000a9d70 0x7f9a001cd4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:57570/0 (socket says 172.21.15.52:57570) 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.979+0000 7f9a04caa640 1 -- 172.21.15.52:0/1157809071 learned_addr learned my addr 172.21.15.52:0/1157809071 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 client.?.objecter ms_handle_connect 0x7f9a000a9d70 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 client.?.objecter ms_handle_connect 0x7f9a0013f650 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 client.?.objecter ms_handle_connect 0x7f9a001457f0 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146158666 0 0) 0x7f9a00138930 con 0x7f9a000a9d70 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.980+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e4004eb0 con 0x7f9a000a9d70 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910386273 0 0) 0x7f9a001cb8b0 con 0x7f9a0013f650 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a00138930 con 0x7f9a0013f650 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194192530 0 0) 0x7f9a001cc7f0 con 0x7f9a001457f0 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:24:24.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a001cb8b0 con 0x7f9a001457f0 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3920643848 0 0) 0x7f99e4004eb0 con 0x7f9a000a9d70 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a001cc7f0 con 0x7f9a000a9d70 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f99f4001950 con 0x7f9a000a9d70 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:24.972 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1560083290 0 0) 0x7f9a001cc7f0 con 0x7f9a000a9d70 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 >> v1:172.21.15.52:6790/0 conn(0x7f9a001457f0 legacy=0x7f9a00138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 >> v1:172.21.15.52:6789/0 conn(0x7f9a0013f650 legacy=0x7f9a00137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 monclient: found mon.b 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.981+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a001ccad0 con 0x7f9a000a9d70 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 10 monclient: _renew_subs 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a001ccdb0 con 0x7f9a000a9d70 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 10 monclient: _renew_subs 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a001cb340 con 0x7f9a000a9d70 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f9a06f0e640 1 librados: init done 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99f40035f0 con 0x7f9a000a9d70 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f99f4003980 con 0x7f9a000a9d70 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:24.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99fd7fa640 10 monclient: dump: 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.982+0000 7f99feffd640 4 set_mon_vals no callback set 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.984+0000 7f9a06f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.984+0000 7f9a06f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.984+0000 7f9a06f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:24.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.984+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0005080 con 0x7f9a000a9d70 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.985+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f99f4003ae0 con 0x7f9a000a9d70 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.985+0000 7f99fffff640 10 monclient: get_auth_request con 0x7f99e406de30 auth_method 0 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.986+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f99f40849f0 con 0x7f9a000a9d70 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.986+0000 7f99fd7fa640 10 client.4378.objecter ms_dispatch 0x7f9a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.986+0000 7f99fd7fa640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.986+0000 7f99fd7fa640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:24.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.986+0000 7f99fd7fa640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.986+0000 7f99fd7fa640 10 client.4378.objecter ms_handle_connect 0x7f99e406de30 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.991+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99f40502d0 con 0x7f9a000a9d70 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.991+0000 7f99fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:24.991+0000 7f99fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.363+0000 7f9a06f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.363+0000 7f9a06f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.363+0000 7f9a06f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.363+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99e0005330 con 0x7f9a000a9d70 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.364+0000 7f99fd7fa640 1 -- 172.21.15.52:0/1157809071 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f99f4052200 con 0x7f9a000a9d70 2024-03-29T11:24:25.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.364+0000 7f99fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.364+0000 7f99fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:25.346 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:25.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 10 librados: watch_flush enter 2024-03-29T11:24:25.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 10 librados: watch_flush exit 2024-03-29T11:24:25.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:24:25.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:24:25.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 >> v1:172.21.15.79:6816/407480584 conn(0x7f99e406de30 legacy=0x7f99e40702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 10 monclient: shutdown 2024-03-29T11:24:25.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:25.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.368+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 >> v1:172.21.15.79:6789/0 conn(0x7f9a000a9d70 legacy=0x7f9a001cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.369+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 shutdown_connections 2024-03-29T11:24:25.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.369+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 >> 172.21.15.52:0/1157809071 conn(0x7f9a00066a00 msgr2=0x7f9a000b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:25.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.369+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 shutdown_connections 2024-03-29T11:24:25.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.369+0000 7f9a06f0e640 1 -- 172.21.15.52:0/1157809071 wait complete. 2024-03-29T11:24:25.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:25.369+0000 7f9a06f0e640 1 librados: shutdown 2024-03-29T11:24:25.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:32.173 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.326+0000 7f03ccf7b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.326+0000 7f03ccf7b640 10 monclient: build_initial_monmap 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.326+0000 7f03ccf7b640 10 monclient: monmap: 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:32.327586+0000 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:32.327586+0000 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.327+0000 7f03ccf7b640 1 Processor -- start 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.328+0000 7f03ccf7b640 1 -- start start 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.328+0000 7f03ccf7b640 10 monclient: init 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.329+0000 7f03ccf7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.329+0000 7f03ccf7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.329+0000 7f03ccf7b640 10 monclient(hunting): picked mon.noname-a con 0x7f03c81457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.329+0000 7f03c5d9b640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f03c813f650 0x7f03c80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:35276/0 (socket says 172.21.15.52:35276) 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03c5d9b640 1 -- 172.21.15.52:0/4290425586 learned_addr learned my addr 172.21.15.52:0/4290425586 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 10 monclient(hunting): picked mon.noname-b con 0x7f03c813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 10 monclient(hunting): picked mon.noname-c con 0x7f03c80a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80aaef0 con 0x7f03c81457f0 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8145bd0 con 0x7f03c80a9d70 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8145eb0 con 0x7f03c813f650 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:32.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.330+0000 7f03ccf7b640 10 monclient(hunting): authenticate will time out at 1202.470872s 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296417889 0 0) 0x7f03c8145eb0 con 0x7f03c813f650 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b4003520 con 0x7f03c813f650 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 135036063 0 0) 0x7f03b4003520 con 0x7f03c813f650 2024-03-29T11:24:32.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 >> v1:172.21.15.79:6789/0 conn(0x7f03c80a9d70 legacy=0x7f03c814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.331+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 >> v1:172.21.15.52:6789/0 conn(0x7f03c81457f0 legacy=0x7f03c813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 1 monclient: found mon.noname-b 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8146340 con 0x7f03c813f650 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: _finish_auth 0 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:32.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f03b8001a10 con 0x7f03c813f650 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.333+0000 7f03c559a640 10 monclient: dump: 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:32.316 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:32.317 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03ccf7b640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03ccf7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03b8001be0 con 0x7f03c813f650 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03ccf7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03ccf7b640 4 set_mon_vals no callback set 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03c559a640 1 -- 172.21.15.52:0/4290425586 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f03b8001ff0 con 0x7f03c813f650 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.334+0000 7f03c559a640 10 monclient: dump: 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:32.318 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 10 monclient: shutdown 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 >> v1:172.21.15.52:6790/0 conn(0x7f03c813f650 legacy=0x7f03c80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 shutdown_connections 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 >> 172.21.15.52:0/4290425586 conn(0x7f03c8066a00 msgr2=0x7f03c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 shutdown_connections 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.335+0000 7f03ccf7b640 1 -- 172.21.15.52:0/4290425586 wait complete. 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.337+0000 7f03ccf7b640 10 monclient: build_initial_monmap 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.337+0000 7f03ccf7b640 10 monclient: monmap: 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 1 librados: starting msgr at 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 1 librados: starting objecter 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 1 Processor -- start 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 1 -- start start 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 1 librados: setting wanted keys 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 1 librados: calling monclient init 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.338+0000 7f03ccf7b640 10 monclient: init 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 10 monclient(hunting): picked mon.noname-b con 0x7f03c80a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 10 monclient(hunting): picked mon.noname-a con 0x7f03c813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 10 monclient(hunting): picked mon.noname-c con 0x7f03c81457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8141fa0 con 0x7f03c813f650 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03c6d9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f03c81457f0 0x7f03c81e29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:36026/0 (socket says 172.21.15.52:36026) 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03c6d9d640 1 -- 172.21.15.52:0/2091878823 learned_addr learned my addr 172.21.15.52:0/2091878823 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03beffd640 10 client.?.objecter ms_handle_connect 0x7f03c81457f0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.339+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80ac530 con 0x7f03c80a9d70 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80ac810 con 0x7f03c81457f0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03ccf7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03ccf7b640 10 monclient(hunting): authenticate will time out at 1202.480153s 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180433543 0 0) 0x7f03c80ac810 con 0x7f03c81457f0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03beffd640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.340+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a8004eb0 con 0x7f03c81457f0 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 client.?.objecter ms_handle_connect 0x7f03c80a9d70 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815833188 0 0) 0x7f03a8004eb0 con 0x7f03c81457f0 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03c80ac810 con 0x7f03c81457f0 2024-03-29T11:24:32.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246862104 0 0) 0x7f03c80ac530 con 0x7f03c80a9d70 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a8004eb0 con 0x7f03c80a9d70 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f03b00018e0 con 0x7f03c81457f0 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): dump: 2024-03-29T11:24:32.331 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 460213220 0 0) 0x7f03c80ac810 con 0x7f03c81457f0 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 >> v1:172.21.15.79:6789/0 conn(0x7f03c80a9d70 legacy=0x7f03c8142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.342+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 >> v1:172.21.15.52:6789/0 conn(0x7f03c813f650 legacy=0x7f03c81370f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:32.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.343+0000 7f03beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 1 monclient: found mon.c 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80ab620 con 0x7f03c81457f0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 10 monclient: _renew_subs 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c813fda0 con 0x7f03c81457f0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 10 monclient: _renew_subs 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:32.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8141ca0 con 0x7f03c81457f0 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03ccf7b640 1 librados: init done 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03b00035f0 con 0x7f03c81457f0 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f03b0001a40 con 0x7f03c81457f0 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03beffd640 10 monclient: dump: 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:32.334 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.344+0000 7f03c4d99640 4 set_mon_vals no callback set 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f03b00039c0 con 0x7f03c81457f0 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03ccf7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03ccf7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03ccf7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ac005080 con 0x7f03c81457f0 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f03b00849f0 con 0x7f03c81457f0 2024-03-29T11:24:32.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03beffd640 10 client.4406.objecter ms_dispatch 0x7f03c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:32.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03beffd640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:32.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03beffd640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:32.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03beffd640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:24:32.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.347+0000 7f03c659c640 10 monclient: get_auth_request con 0x7f03a806dd00 auth_method 0 2024-03-29T11:24:32.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.348+0000 7f03beffd640 10 client.4406.objecter ms_handle_connect 0x7f03a806dd00 2024-03-29T11:24:32.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.355+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03b00502d0 con 0x7f03c81457f0 2024-03-29T11:24:32.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.355+0000 7f03beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.355+0000 7f03beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:32.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.672+0000 7f03ccf7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.672+0000 7f03ccf7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.672+0000 7f03ccf7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:32.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.672+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03ac005330 con 0x7f03c81457f0 2024-03-29T11:24:32.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.673+0000 7f03beffd640 1 -- 172.21.15.52:0/2091878823 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f03b0052200 con 0x7f03c81457f0 2024-03-29T11:24:32.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.673+0000 7f03beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.673+0000 7f03beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:32.655 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:32.656 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:32.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.676+0000 7f03ccf7b640 10 librados: watch_flush enter 2024-03-29T11:24:32.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.676+0000 7f03ccf7b640 10 librados: watch_flush exit 2024-03-29T11:24:32.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.676+0000 7f03ccf7b640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:24:32.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.676+0000 7f03ccf7b640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.677+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 >> v1:172.21.15.79:6816/407480584 conn(0x7f03a806dd00 legacy=0x7f03a80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.677+0000 7f03ccf7b640 10 monclient: shutdown 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.677+0000 7f03ccf7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.677+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 >> v1:172.21.15.52:6790/0 conn(0x7f03c81457f0 legacy=0x7f03c81e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.677+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 shutdown_connections 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.677+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 >> 172.21.15.52:0/2091878823 conn(0x7f03c8066a00 msgr2=0x7f03c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.678+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 shutdown_connections 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.678+0000 7f03ccf7b640 1 -- 172.21.15.52:0/2091878823 wait complete. 2024-03-29T11:24:32.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:32.678+0000 7f03ccf7b640 1 librados: shutdown 2024-03-29T11:24:32.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:39.480 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.645+0000 7f0643aa8640 10 monclient: get_monmap_and_config 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.645+0000 7f0643aa8640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.645+0000 7f0643aa8640 10 monclient: monmap: 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:39.646491+0000 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:39.646491+0000 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:39.628 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:39.629 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:39.629 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:39.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.646+0000 7f0643aa8640 1 Processor -- start 2024-03-29T11:24:39.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.646+0000 7f0643aa8640 1 -- start start 2024-03-29T11:24:39.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.646+0000 7f0643aa8640 10 monclient: init 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.647+0000 7f0643aa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.647+0000 7f0643aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.647+0000 7f0643aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f063c0ae250 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.647+0000 7f0643aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f063c13f970 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.647+0000 7f0643aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f063c142800 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.647+0000 7f0643aa8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063c0afe00 con 0x7f063c0ae250 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.648+0000 7f0643aa8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063c142c10 con 0x7f063c13f970 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.648+0000 7f0643aa8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063c0afb20 con 0x7f063c142800 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.648+0000 7f0643aa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.648+0000 7f0643aa8640 10 monclient(hunting): authenticate will time out at 1209.788092s 2024-03-29T11:24:39.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.648+0000 7f0642045640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f063c142800 0x7f063c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:48924/0 (socket says 172.21.15.52:48924) 2024-03-29T11:24:39.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.649+0000 7f0642045640 1 -- 172.21.15.52:0/2221650748 learned_addr learned my addr 172.21.15.52:0/2221650748 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48399027 0 0) 0x7f063c0afe00 con 0x7f063c0ae250 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0628003520 con 0x7f063c0ae250 2024-03-29T11:24:39.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1025656296 0 0) 0x7f0628003520 con 0x7f063c0ae250 2024-03-29T11:24:39.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 >> v1:172.21.15.52:6790/0 conn(0x7f063c142800 legacy=0x7f063c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:39.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.651+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 >> v1:172.21.15.79:6789/0 conn(0x7f063c13f970 legacy=0x7f063c1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:39.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 1 monclient: found mon.noname-a 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c0aab80 con 0x7f063c0ae250 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0634001a10 con 0x7f063c0ae250 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0643aa8640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0643aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0640842640 10 monclient: dump: 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:39.635 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:39.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.652+0000 7f0643aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0634001be0 con 0x7f063c0ae250 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0640842640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0640842640 1 -- 172.21.15.52:0/2221650748 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0634002030 con 0x7f063c0ae250 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0640842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0640842640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0640842640 10 monclient: dump: 2024-03-29T11:24:39.639 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0643aa8640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0643aa8640 4 set_mon_vals no callback set 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0643aa8640 10 monclient: shutdown 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0643aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.653+0000 7f0643aa8640 1 -- 172.21.15.52:0/2221650748 >> v1:172.21.15.52:6789/0 conn(0x7f063c0ae250 legacy=0x7f063c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.654+0000 7f0643aa8640 1 -- 172.21.15.52:0/2221650748 shutdown_connections 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.654+0000 7f0643aa8640 1 -- 172.21.15.52:0/2221650748 >> 172.21.15.52:0/2221650748 conn(0x7f063c0b35e0 msgr2=0x7f063c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.655+0000 7f0643aa8640 1 -- 172.21.15.52:0/2221650748 shutdown_connections 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.655+0000 7f0643aa8640 1 -- 172.21.15.52:0/2221650748 wait complete. 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.656+0000 7f0643aa8640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.656+0000 7f0643aa8640 10 monclient: monmap: 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:39.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.656+0000 7f0643aa8640 1 librados: starting msgr at 2024-03-29T11:24:39.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.656+0000 7f0643aa8640 1 librados: starting objecter 2024-03-29T11:24:39.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.656+0000 7f0643aa8640 1 Processor -- start 2024-03-29T11:24:39.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.657+0000 7f0643aa8640 1 -- start start 2024-03-29T11:24:39.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.657+0000 7f0643aa8640 1 librados: setting wanted keys 2024-03-29T11:24:39.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.657+0000 7f0643aa8640 1 librados: calling monclient init 2024-03-29T11:24:39.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.657+0000 7f0643aa8640 10 monclient: init 2024-03-29T11:24:39.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f063c0ae250 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f063c13f970 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f063c142800 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063c06edd0 con 0x7f063c0ae250 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063c145b50 con 0x7f063c142800 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063c145630 con 0x7f063c13f970 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.658+0000 7f0643aa8640 10 monclient(hunting): authenticate will time out at 1209.798794s 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.661+0000 7f0641043640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f063c13f970 0x7f063c138c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49506/0 (socket says 172.21.15.52:49506) 2024-03-29T11:24:39.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.661+0000 7f0641043640 1 -- 172.21.15.52:0/2105737865 learned_addr learned my addr 172.21.15.52:0/2105737865 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 client.?.objecter ms_handle_connect 0x7f063c13f970 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148836954 0 0) 0x7f063c145630 con 0x7f063c13f970 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0620004eb0 con 0x7f063c13f970 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 client.?.objecter ms_handle_connect 0x7f063c142800 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 client.?.objecter ms_handle_connect 0x7f063c0ae250 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665093911 0 0) 0x7f063c145b50 con 0x7f063c142800 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.664+0000 7f06327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c145630 con 0x7f063c142800 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754194138 0 0) 0x7f0620004eb0 con 0x7f063c13f970 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f063c145b50 con 0x7f063c13f970 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f062c0017f0 con 0x7f063c13f970 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): dump: 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:39.652 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432084380 0 0) 0x7f063c06edd0 con 0x7f063c0ae250 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0620004eb0 con 0x7f063c0ae250 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1061873695 0 0) 0x7f063c145b50 con 0x7f063c13f970 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 >> v1:172.21.15.79:6789/0 conn(0x7f063c142800 legacy=0x7f063c1d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 >> v1:172.21.15.52:6789/0 conn(0x7f063c0ae250 legacy=0x7f063c144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 monclient: found mon.c 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c145270 con 0x7f063c13f970 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.665+0000 7f06327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 10 monclient: _renew_subs 2024-03-29T11:24:39.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063c139330 con 0x7f063c13f970 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f062c001950 con 0x7f063c13f970 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 10 monclient: _renew_subs 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f063c0ac270 con 0x7f063c13f970 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.666+0000 7f0643aa8640 1 librados: init done 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f0643aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f0643aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f0643aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063c0ae630 con 0x7f063c13f970 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f06327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f062c004aa0 con 0x7f063c13f970 2024-03-29T11:24:39.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f06327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f06327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f06327fc640 10 monclient: dump: 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:39.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.668+0000 7f0633fff640 4 set_mon_vals no callback set 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.669+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f062c01aa70 con 0x7f063c13f970 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.670+0000 7f0641844640 10 monclient: get_auth_request con 0x7f062006de80 auth_method 0 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.673+0000 7f06327fc640 10 client.4409.objecter ms_handle_connect 0x7f062006de80 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.673+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f062c04ff10 con 0x7f063c13f970 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.673+0000 7f06327fc640 10 client.4409.objecter ms_dispatch 0x7f063c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.673+0000 7f06327fc640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.673+0000 7f06327fc640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:39.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.673+0000 7f06327fc640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:24:39.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.677+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f063c0ae630 con 0x7f063c13f970 2024-03-29T11:24:39.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.677+0000 7f06327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.677+0000 7f06327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:39.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.993+0000 7f0643aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.993+0000 7f0643aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.993+0000 7f0643aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:24:39.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.993+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f063c0ae630 con 0x7f063c13f970 2024-03-29T11:24:39.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.995+0000 7f06327fc640 1 -- 172.21.15.52:0/2105737865 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f063c0ae630 con 0x7f063c13f970 2024-03-29T11:24:39.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.995+0000 7f06327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.995+0000 7f06327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:39.977 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:39.977 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.998+0000 7f0643aa8640 10 librados: watch_flush enter 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.998+0000 7f0643aa8640 10 librados: watch_flush exit 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.998+0000 7f0643aa8640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.998+0000 7f0643aa8640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 >> v1:172.21.15.79:6816/407480584 conn(0x7f062006de80 legacy=0x7f0620070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 10 monclient: shutdown 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 >> v1:172.21.15.52:6790/0 conn(0x7f063c13f970 legacy=0x7f063c138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 shutdown_connections 2024-03-29T11:24:39.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 >> 172.21.15.52:0/2105737865 conn(0x7f063c0b35e0 msgr2=0x7f063c0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 shutdown_connections 2024-03-29T11:24:39.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 -- 172.21.15.52:0/2105737865 wait complete. 2024-03-29T11:24:39.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:39.999+0000 7f0643aa8640 1 librados: shutdown 2024-03-29T11:24:39.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:46.804 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:46.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.980+0000 7f187f34a640 10 monclient: get_monmap_and_config 2024-03-29T11:24:46.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.980+0000 7f187f34a640 10 monclient: build_initial_monmap 2024-03-29T11:24:46.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.980+0000 7f187f34a640 10 monclient: monmap: 2024-03-29T11:24:46.963 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:46.965 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:46.965 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:46.981032+0000 2024-03-29T11:24:46.965 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:46.981032+0000 2024-03-29T11:24:46.965 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:46.965 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:46.966 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:46.966 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:46.966 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:46.966 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:46.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.982+0000 7f187f34a640 1 Processor -- start 2024-03-29T11:24:46.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.982+0000 7f187f34a640 1 -- start start 2024-03-29T11:24:46.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.982+0000 7f187f34a640 10 monclient: init 2024-03-29T11:24:46.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:46.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:46.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient(hunting): picked mon.noname-b con 0x7f18780a9750 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient(hunting): picked mon.noname-a con 0x7f18780ae220 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient(hunting): picked mon.noname-c con 0x7f187813b9d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1878142490 con 0x7f18780ae220 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780ac130 con 0x7f187813b9d0 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780ac410 con 0x7f18780a9750 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:46.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.983+0000 7f187f34a640 10 monclient(hunting): authenticate will time out at 1217.123947s 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187db47640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f18780ae220 0x7f187813b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46703/0 (socket says 172.21.15.52:46703) 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187db47640 1 -- 172.21.15.52:0/1710454809 learned_addr learned my addr 172.21.15.52:0/1710454809 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307669278 0 0) 0x7f1878142490 con 0x7f18780ae220 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:46.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1864003520 con 0x7f18780ae220 2024-03-29T11:24:46.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 769989253 0 0) 0x7f1864003520 con 0x7f18780ae220 2024-03-29T11:24:46.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.987+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 >> v1:172.21.15.52:6790/0 conn(0x7f18780a9750 legacy=0x7f1878141d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:46.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 >> v1:172.21.15.79:6789/0 conn(0x7f187813b9d0 legacy=0x7f1878149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:46.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: _finish_hunting 0 2024-03-29T11:24:46.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 1 monclient: found mon.noname-a 2024-03-29T11:24:46.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:46.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:24:46.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780afa40 con 0x7f18780ae220 2024-03-29T11:24:46.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: _finish_auth 0 2024-03-29T11:24:46.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: _check_auth_tickets 2024-03-29T11:24:46.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:46.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1874001690 con 0x7f18780ae220 2024-03-29T11:24:46.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187d346640 10 monclient: dump: 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.977 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:46.978 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:46.978 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:46.978 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:46.978 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:46.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187f34a640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:24:46.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.988+0000 7f187f34a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:46.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18740017f0 con 0x7f18780ae220 2024-03-29T11:24:46.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187d346640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:46.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187d346640 1 -- 172.21.15.52:0/1710454809 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1874001c40 con 0x7f18780ae220 2024-03-29T11:24:46.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187d346640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187d346640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:46.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187d346640 10 monclient: dump: 2024-03-29T11:24:46.981 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:46.982 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:46.982 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:46.982 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:46.982 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.982 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:46.982 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187f34a640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.993+0000 7f187f34a640 4 set_mon_vals no callback set 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 10 monclient: shutdown 2024-03-29T11:24:46.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:46.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 1 -- 172.21.15.52:0/1710454809 >> v1:172.21.15.52:6789/0 conn(0x7f18780ae220 legacy=0x7f187813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 1 -- 172.21.15.52:0/1710454809 shutdown_connections 2024-03-29T11:24:46.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 1 -- 172.21.15.52:0/1710454809 >> 172.21.15.52:0/1710454809 conn(0x7f18780673e0 msgr2=0x7f18780b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:46.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 1 -- 172.21.15.52:0/1710454809 shutdown_connections 2024-03-29T11:24:46.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.994+0000 7f187f34a640 1 -- 172.21.15.52:0/1710454809 wait complete. 2024-03-29T11:24:46.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.995+0000 7f187f34a640 10 monclient: build_initial_monmap 2024-03-29T11:24:46.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.995+0000 7f187f34a640 10 monclient: monmap: 2024-03-29T11:24:46.985 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:46.985 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:46.985 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:46.985 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:46.985 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:46.986 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:46.986 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:46.986 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:46.986 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:46.986 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:46.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 1 librados: starting msgr at 2024-03-29T11:24:46.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 1 librados: starting objecter 2024-03-29T11:24:46.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 1 Processor -- start 2024-03-29T11:24:46.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 1 -- start start 2024-03-29T11:24:46.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 1 librados: setting wanted keys 2024-03-29T11:24:46.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 1 librados: calling monclient init 2024-03-29T11:24:46.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.996+0000 7f187f34a640 10 monclient: init 2024-03-29T11:24:46.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:46.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:46.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient(hunting): picked mon.noname-a con 0x7f18780a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:46.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient(hunting): picked mon.noname-b con 0x7f18780ae220 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:46.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient(hunting): picked mon.noname-c con 0x7f187813b9d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:46.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780b28e0 con 0x7f18780a9750 2024-03-29T11:24:46.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780b2b50 con 0x7f18780ae220 2024-03-29T11:24:46.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1878139ff0 con 0x7f187813b9d0 2024-03-29T11:24:46.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:46.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:46.997+0000 7f187f34a640 10 monclient(hunting): authenticate will time out at 1217.137417s 2024-03-29T11:24:46.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.002+0000 7f187db47640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f18780ae220 0x7f1878131780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:46057/0 (socket says 172.21.15.52:46057) 2024-03-29T11:24:46.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.002+0000 7f187db47640 1 -- 172.21.15.52:0/1691515006 learned_addr learned my addr 172.21.15.52:0/1691515006 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:46.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.002+0000 7f186effd640 10 client.?.objecter ms_handle_connect 0x7f18780ae220 2024-03-29T11:24:46.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.002+0000 7f186effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:46.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205941161 0 0) 0x7f18780b2b50 con 0x7f18780ae220 2024-03-29T11:24:46.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:24:46.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c004eb0 con 0x7f18780ae220 2024-03-29T11:24:46.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 client.?.objecter ms_handle_connect 0x7f187813b9d0 2024-03-29T11:24:46.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:46.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f187eb49640 1 -- 172.21.15.52:0/1691515006 >> v1:172.21.15.52:6790/0 conn(0x7f187813b9d0 legacy=0x7f18781ef710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:24:46.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f187eb49640 1 -- 172.21.15.52:0/1691515006 >> v1:172.21.15.52:6790/0 conn(0x7f187813b9d0 legacy=0x7f18781ef710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:24:46.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f187eb49640 1 --1- 172.21.15.52:0/1691515006 >> v1:172.21.15.52:6790/0 conn(0x7f187813b9d0 0x7f18781ef710 :-1 s=OPENED pgs=459087 cs=1 l=1).handle_message read tag failed 2024-03-29T11:24:46.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f187eb49640 1 --1- 172.21.15.52:0/1691515006 >> v1:172.21.15.52:6790/0 conn(0x7f187813b9d0 0x7f18781ef710 :-1 s=OPENED pgs=459087 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:24:46.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.52:6790/0 2024-03-29T11:24:46.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2134169434 0 0) 0x7f185c004eb0 con 0x7f18780ae220 2024-03-29T11:24:46.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f185c005300 con 0x7f18780ae220 2024-03-29T11:24:46.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1874001bc0 con 0x7f18780ae220 2024-03-29T11:24:46.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.003+0000 7f186effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:46.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient(hunting): dump: 2024-03-29T11:24:46.998 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:46.999 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:46.999 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:46.999 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:46.999 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.999 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:46.999 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:47.000 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:47.000 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:47.000 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:47.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:47.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 client.?.objecter ms_handle_connect 0x7f18780a9750 2024-03-29T11:24:47.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1277968442 0 0) 0x7f185c005300 con 0x7f18780ae220 2024-03-29T11:24:47.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 >> v1:172.21.15.52:6790/0 conn(0x7f187813b9d0 legacy=0x7f18781ef710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:24:47.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 >> v1:172.21.15.52:6789/0 conn(0x7f18780a9750 legacy=0x7f187813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 1 monclient: found mon.b 2024-03-29T11:24:47.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:47.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878139d60 con 0x7f18780ae220 2024-03-29T11:24:47.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f186effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 10 monclient: _renew_subs 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18780b17f0 con 0x7f18780ae220 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 10 monclient: _renew_subs 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:47.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18780b1410 con 0x7f18780ae220 2024-03-29T11:24:47.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.004+0000 7f187f34a640 1 librados: init done 2024-03-29T11:24:47.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.005+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1874004bc0 con 0x7f18780ae220 2024-03-29T11:24:47.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.005+0000 7f186effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.005+0000 7f187cb45640 4 set_mon_vals no callback set 2024-03-29T11:24:47.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.006+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1874004ed0 con 0x7f18780ae220 2024-03-29T11:24:47.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.006+0000 7f186effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.006+0000 7f186effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:47.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.006+0000 7f186effd640 10 monclient: dump: 2024-03-29T11:24:47.005 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:47.005 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:47.006 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:47.007 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:47.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.007+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f187401aea0 con 0x7f18780ae220 2024-03-29T11:24:47.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.007+0000 7f187e348640 10 monclient: get_auth_request con 0x7f185c06e0a0 auth_method 0 2024-03-29T11:24:47.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f18740547d0 con 0x7f18780ae220 2024-03-29T11:24:47.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f186effd640 10 client.4387.objecter ms_dispatch 0x7f1878000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:47.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f186effd640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:47.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f186effd640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:47.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f186effd640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-29T11:24:47.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f187f34a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f187f34a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f187f34a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:47.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f186effd640 10 client.4387.objecter ms_handle_connect 0x7f185c06e0a0 2024-03-29T11:24:47.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.008+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860005080 con 0x7f18780ae220 2024-03-29T11:24:47.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.016+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1874050a70 con 0x7f18780ae220 2024-03-29T11:24:47.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.016+0000 7f186effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.016+0000 7f186effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:47.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.340+0000 7f187f34a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.340+0000 7f187f34a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.340+0000 7f187f34a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:24:47.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.340+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1860005330 con 0x7f18780ae220 2024-03-29T11:24:47.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.343+0000 7f186effd640 1 -- 172.21.15.52:0/1691515006 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f1874050a70 con 0x7f18780ae220 2024-03-29T11:24:47.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.344+0000 7f186effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.344+0000 7f186effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:47.327 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:47.328 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:47.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.347+0000 7f187f34a640 10 librados: watch_flush enter 2024-03-29T11:24:47.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.347+0000 7f187f34a640 10 librados: watch_flush exit 2024-03-29T11:24:47.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.347+0000 7f187f34a640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-29T11:24:47.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.347+0000 7f187f34a640 10 client.4387.objecter successfully canceled tick 2024-03-29T11:24:47.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 >> v1:172.21.15.79:6816/407480584 conn(0x7f185c06e0a0 legacy=0x7f185c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 10 monclient: shutdown 2024-03-29T11:24:47.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:47.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 >> v1:172.21.15.79:6789/0 conn(0x7f18780ae220 legacy=0x7f1878131780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 shutdown_connections 2024-03-29T11:24:47.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 >> 172.21.15.52:0/1691515006 conn(0x7f18780673e0 msgr2=0x7f18780b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:47.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 shutdown_connections 2024-03-29T11:24:47.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 -- 172.21.15.52:0/1691515006 wait complete. 2024-03-29T11:24:47.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:47.348+0000 7f187f34a640 1 librados: shutdown 2024-03-29T11:24:47.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:54.155 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.319+0000 7fd359efb640 10 monclient: get_monmap_and_config 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.319+0000 7fd359efb640 10 monclient: build_initial_monmap 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.319+0000 7fd359efb640 10 monclient: monmap: 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:24:54.320153+0000 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:24:54.320153+0000 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:54.302 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:54.303 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:24:54.303 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:24:54.303 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:54.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.321+0000 7fd359efb640 1 Processor -- start 2024-03-29T11:24:54.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.321+0000 7fd359efb640 1 -- start start 2024-03-29T11:24:54.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.321+0000 7fd359efb640 10 monclient: init 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient(hunting): picked mon.noname-c con 0x7fd3541422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient(hunting): picked mon.noname-a con 0x7fd354142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient(hunting): picked mon.noname-b con 0x7fd3540b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354142c80 con 0x7fd354142690 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3540aae40 con 0x7fd3541422b0 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35413fbd0 con 0x7fd3540b17d0 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:54.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.322+0000 7fd359efb640 10 monclient(hunting): authenticate will time out at 1224.462319s 2024-03-29T11:24:54.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.327+0000 7fd352ffd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fd354142690 0x7fd3540b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:37235/0 (socket says 172.21.15.52:37235) 2024-03-29T11:24:54.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.327+0000 7fd352ffd640 1 -- 172.21.15.52:0/4110073463 learned_addr learned my addr 172.21.15.52:0/4110073463 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.328+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959217568 0 0) 0x7fd354142c80 con 0x7fd354142690 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.328+0000 7fd3527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.328+0000 7fd3527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.328+0000 7fd3527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.328+0000 7fd3527fc640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.328+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c003520 con 0x7fd354142690 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156420084 0 0) 0x7fd33c003520 con 0x7fd354142690 2024-03-29T11:24:54.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 >> v1:172.21.15.52:6790/0 conn(0x7fd3540b17d0 legacy=0x7fd3540b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 >> v1:172.21.15.79:6789/0 conn(0x7fd3541422b0 legacy=0x7fd35413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 1 monclient: found mon.noname-a 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354151940 con 0x7fd354142690 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd359efb640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd359efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fd34c0019f0 con 0x7fd354142690 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd3527fc640 10 monclient: dump: 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.329+0000 7fd359efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.330+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd34c001c10 con 0x7fd354142690 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.330+0000 7fd3527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:54.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.330+0000 7fd359efb640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.330+0000 7fd359efb640 4 set_mon_vals no callback set 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd359efb640 10 monclient: shutdown 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd359efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd359efb640 1 -- 172.21.15.52:0/4110073463 >> v1:172.21.15.52:6789/0 conn(0x7fd354142690 legacy=0x7fd3540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd3527fc640 1 -- 172.21.15.52:0/4110073463 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fd34c0019f0 con 0x7fd354142690 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd3527fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd359efb640 1 -- 172.21.15.52:0/4110073463 shutdown_connections 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd359efb640 1 -- 172.21.15.52:0/4110073463 >> 172.21.15.52:0/4110073463 conn(0x7fd354066a00 msgr2=0x7fd354066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.331+0000 7fd359efb640 1 -- 172.21.15.52:0/4110073463 shutdown_connections 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.333+0000 7fd359efb640 1 -- 172.21.15.52:0/4110073463 wait complete. 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.334+0000 7fd359efb640 10 monclient: build_initial_monmap 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.334+0000 7fd359efb640 10 monclient: monmap: 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:54.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.334+0000 7fd359efb640 1 librados: starting msgr at 2024-03-29T11:24:54.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.334+0000 7fd359efb640 1 librados: starting objecter 2024-03-29T11:24:54.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.334+0000 7fd359efb640 1 Processor -- start 2024-03-29T11:24:54.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.335+0000 7fd359efb640 1 -- start start 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.335+0000 7fd359efb640 1 librados: setting wanted keys 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.335+0000 7fd359efb640 1 librados: calling monclient init 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.335+0000 7fd359efb640 10 monclient: init 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient(hunting): picked mon.noname-c con 0x7fd3540b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient(hunting): picked mon.noname-b con 0x7fd3541422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient(hunting): picked mon.noname-a con 0x7fd354142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354071800 con 0x7fd354142690 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35413c750 con 0x7fd3541422b0 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35406fe90 con 0x7fd3540b17d0 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:54.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.336+0000 7fd359efb640 10 monclient(hunting): authenticate will time out at 1224.476275s 2024-03-29T11:24:54.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd353fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fd354142690 0x7fd35413c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:37349/0 (socket says 172.21.15.52:37349) 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd353fff640 1 -- 172.21.15.52:0/2647739308 learned_addr learned my addr 172.21.15.52:0/2647739308 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 10 client.?.objecter ms_handle_connect 0x7fd354142690 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090940910 0 0) 0x7fd354071800 con 0x7fd354142690 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340004eb0 con 0x7fd354142690 2024-03-29T11:24:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.339+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988315480 0 0) 0x7fd340004eb0 con 0x7fd354142690 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd354071800 con 0x7fd354142690 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fd348001970 con 0x7fd354142690 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 10 monclient(hunting): dump: 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:54.328 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3788368822 0 0) 0x7fd354071800 con 0x7fd354142690 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.340+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 >> v1:172.21.15.52:6790/0 conn(0x7fd3540b17d0 legacy=0x7fd35406de90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.341+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 >> v1:172.21.15.79:6789/0 conn(0x7fd3541422b0 legacy=0x7fd35413b930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 1 monclient: found mon.a 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354070170 con 0x7fd354142690 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd33bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 10 monclient: _renew_subs 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:54.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd354071320 con 0x7fd354142690 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 10 monclient: _renew_subs 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.342+0000 7fd359efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.343+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd354070900 con 0x7fd354142690 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.343+0000 7fd359efb640 1 librados: init done 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.344+0000 7fd359efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.344+0000 7fd359efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.344+0000 7fd359efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:54.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.344+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3540719b0 con 0x7fd354142690 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd348001ad0 con 0x7fd354142690 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd33bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fd348003a60 con 0x7fd354142690 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd33bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd33bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd33bfff640 10 monclient: dump: 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:24:54.331 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:24:54.332 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:24:54.332 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:24:54.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.345+0000 7fd3517fa640 4 set_mon_vals no callback set 2024-03-29T11:24:54.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.349+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fd348003bc0 con 0x7fd354142690 2024-03-29T11:24:54.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.349+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fd348084a50 con 0x7fd354142690 2024-03-29T11:24:54.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.349+0000 7fd33bfff640 10 client.4416.objecter ms_dispatch 0x7fd354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:54.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.349+0000 7fd33bfff640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:54.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.349+0000 7fd33bfff640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:54.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.349+0000 7fd33bfff640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:24:54.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.350+0000 7fd3537fe640 10 monclient: get_auth_request con 0x7fd34006dd60 auth_method 0 2024-03-29T11:24:54.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.353+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd348050330 con 0x7fd354142690 2024-03-29T11:24:54.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.353+0000 7fd33bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.353+0000 7fd33bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:54.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.354+0000 7fd33bfff640 10 client.4416.objecter ms_handle_connect 0x7fd34006dd60 2024-03-29T11:24:54.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.673+0000 7fd359efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.673+0000 7fd359efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.673+0000 7fd359efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:24:54.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.673+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd354075f10 con 0x7fd354142690 2024-03-29T11:24:54.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.675+0000 7fd33bfff640 1 -- 172.21.15.52:0/2647739308 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fd348052260 con 0x7fd354142690 2024-03-29T11:24:54.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.675+0000 7fd33bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.675+0000 7fd33bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:54.660 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:24:54.660 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.678+0000 7fd359efb640 10 librados: watch_flush enter 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 10 librados: watch_flush exit 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 >> v1:172.21.15.79:6816/407480584 conn(0x7fd34006dd60 legacy=0x7fd340070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 10 monclient: shutdown 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 >> v1:172.21.15.52:6789/0 conn(0x7fd354142690 legacy=0x7fd35413c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 shutdown_connections 2024-03-29T11:24:54.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.679+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 >> 172.21.15.52:0/2647739308 conn(0x7fd354066a00 msgr2=0x7fd35413da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.680+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 shutdown_connections 2024-03-29T11:24:54.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.680+0000 7fd359efb640 1 -- 172.21.15.52:0/2647739308 wait complete. 2024-03-29T11:24:54.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:24:54.680+0000 7fd359efb640 1 librados: shutdown 2024-03-29T11:24:54.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:01.490 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:01.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.657+0000 7f74c7594640 10 monclient: get_monmap_and_config 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.657+0000 7f74c7594640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.657+0000 7f74c7594640 10 monclient: monmap: 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:01.659278+0000 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:01.659278+0000 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:01.642 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:01.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.658+0000 7f74c7594640 1 Processor -- start 2024-03-29T11:25:01.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.658+0000 7f74c7594640 1 -- start start 2024-03-29T11:25:01.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.658+0000 7f74c7594640 10 monclient: init 2024-03-29T11:25:01.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.659+0000 7f74c7594640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.660+0000 7f74c7594640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.662+0000 7f74c7594640 10 monclient(hunting): picked mon.noname-a con 0x7f74b8034850 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.662+0000 7f74c7594640 10 monclient(hunting): picked mon.noname-b con 0x7f74b802a640 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.662+0000 7f74c7594640 10 monclient(hunting): picked mon.noname-c con 0x7f74b80327d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.664+0000 7f74c6592640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f74b8034850 0x7f74b80c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:60491/0 (socket says 172.21.15.52:60491) 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.665+0000 7f74c6592640 1 -- 172.21.15.52:0/2063304748 learned_addr learned my addr 172.21.15.52:0/2063304748 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.665+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b802c820 con 0x7f74b8034850 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.666+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b802a120 con 0x7f74b80327d0 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.666+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b80c75b0 con 0x7f74b802a640 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.666+0000 7f74c7594640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.666+0000 7f74c7594640 10 monclient(hunting): authenticate will time out at 1231.807333s 2024-03-29T11:25:01.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989837522 0 0) 0x7f74b80c75b0 con 0x7f74b802a640 2024-03-29T11:25:01.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:25:01.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ac003520 con 0x7f74b802a640 2024-03-29T11:25:01.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 330011897 0 0) 0x7f74ac003520 con 0x7f74b802a640 2024-03-29T11:25:01.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 >> v1:172.21.15.79:6789/0 conn(0x7f74b80327d0 legacy=0x7f74b80cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 >> v1:172.21.15.52:6789/0 conn(0x7f74b8034850 legacy=0x7f74b80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 1 monclient: found mon.noname-b 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.669+0000 7f74c5590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b8033110 con 0x7f74b802a640 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f74bc0015c0 con 0x7f74b802a640 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c7594640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c7594640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: dump: 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c7594640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74bc001850 con 0x7f74b802a640 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c5590640 1 -- 172.21.15.52:0/2063304748 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f74bc001be0 con 0x7f74b802a640 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c7594640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.670+0000 7f74c7594640 4 set_mon_vals no callback set 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c5590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c5590640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c5590640 10 monclient: dump: 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:01.658 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:01.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c7594640 10 monclient: shutdown 2024-03-29T11:25:01.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c7594640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 >> v1:172.21.15.52:6790/0 conn(0x7f74b802a640 legacy=0x7f74b80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 shutdown_connections 2024-03-29T11:25:01.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.671+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 >> 172.21.15.52:0/2063304748 conn(0x7f74b80058f0 msgr2=0x7f74b8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.675+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 shutdown_connections 2024-03-29T11:25:01.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.676+0000 7f74c7594640 1 -- 172.21.15.52:0/2063304748 wait complete. 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.677+0000 7f74c7594640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.677+0000 7f74c7594640 10 monclient: monmap: 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:01.663 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.677+0000 7f74c7594640 1 librados: starting msgr at 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.677+0000 7f74c7594640 1 librados: starting objecter 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.677+0000 7f74c7594640 1 Processor -- start 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.678+0000 7f74c7594640 1 -- start start 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.678+0000 7f74c7594640 1 librados: setting wanted keys 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.678+0000 7f74c7594640 1 librados: calling monclient init 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.678+0000 7f74c7594640 10 monclient: init 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.678+0000 7f74c7594640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.678+0000 7f74c7594640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 10 monclient(hunting): picked mon.noname-a con 0x7f74b802a640 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 10 monclient(hunting): picked mon.noname-b con 0x7f74b80327d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 10 monclient(hunting): picked mon.noname-c con 0x7f74b8034850 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b814f960 con 0x7f74b802a640 2024-03-29T11:25:01.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b80c2e10 con 0x7f74b80327d0 2024-03-29T11:25:01.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b814f440 con 0x7f74b8034850 2024-03-29T11:25:01.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.679+0000 7f74c7594640 10 monclient(hunting): authenticate will time out at 1231.820330s 2024-03-29T11:25:01.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.682+0000 7f74c6d93640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f74b8034850 0x7f74b8168000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:54485/0 (socket says 172.21.15.52:54485) 2024-03-29T11:25:01.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.682+0000 7f74c6d93640 1 -- 172.21.15.52:0/1213332576 learned_addr learned my addr 172.21.15.52:0/1213332576 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:01.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 10 client.?.objecter ms_handle_connect 0x7f74b8034850 2024-03-29T11:25:01.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571149082 0 0) 0x7f74b814f440 con 0x7f74b8034850 2024-03-29T11:25:01.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:25:01.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.683+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a4004eb0 con 0x7f74b8034850 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2823084904 0 0) 0x7f74a4004eb0 con 0x7f74b8034850 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b814f440 con 0x7f74b8034850 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f74b0003fd0 con 0x7f74b8034850 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:01.669 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:01.670 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:01.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.684+0000 7f74b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:01.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.685+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939908707 0 0) 0x7f74b814f440 con 0x7f74b8034850 2024-03-29T11:25:01.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.685+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 >> v1:172.21.15.79:6789/0 conn(0x7f74b80327d0 legacy=0x7f74b80c3510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:01.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.686+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 >> v1:172.21.15.52:6789/0 conn(0x7f74b802a640 legacy=0x7f74b80baef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.686+0000 7f74b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.686+0000 7f74b6ffd640 1 monclient: found mon.c 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.686+0000 7f74b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b80c45e0 con 0x7f74b8034850 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 10 client.?.objecter ms_handle_connect 0x7f74b802a640 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74c7594640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74c7594640 10 monclient: _renew_subs 2024-03-29T11:25:01.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74c7594640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b80c47d0 con 0x7f74b8034850 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.687+0000 7f74c7594640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74c7594640 10 monclient: _renew_subs 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74c7594640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b80bb7e0 con 0x7f74b8034850 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74c7594640 1 librados: init done 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74b0004130 con 0x7f74b8034850 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f74b0001170 con 0x7f74b8034850 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:01.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.688+0000 7f74b6ffd640 10 monclient: dump: 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:01.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.689+0000 7f74c4d8f640 4 set_mon_vals no callback set 2024-03-29T11:25:01.677 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.691+0000 7f74c7594640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.677 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.691+0000 7f74c7594640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.677 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.691+0000 7f74c7594640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:01.677 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.691+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a8005080 con 0x7f74b8034850 2024-03-29T11:25:01.677 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.693+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f74b00013d0 con 0x7f74b8034850 2024-03-29T11:25:01.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.694+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f74b0084a20 con 0x7f74b8034850 2024-03-29T11:25:01.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.694+0000 7f74b6ffd640 10 client.4421.objecter ms_dispatch 0x7f74c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:01.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.694+0000 7f74b6ffd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:01.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.694+0000 7f74b6ffd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:01.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.694+0000 7f74b6ffd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:25:01.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.695+0000 7f74c6592640 10 monclient: get_auth_request con 0x7f74a406dc10 auth_method 0 2024-03-29T11:25:01.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.695+0000 7f74b6ffd640 10 client.4421.objecter ms_handle_connect 0x7f74a406dc10 2024-03-29T11:25:01.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.701+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74b0050300 con 0x7f74b8034850 2024-03-29T11:25:01.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.701+0000 7f74b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:01.701+0000 7f74b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:02.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.038+0000 7f74c7594640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.038+0000 7f74c7594640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.038+0000 7f74c7594640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:02.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.038+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74a8005330 con 0x7f74b8034850 2024-03-29T11:25:02.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.040+0000 7f74b6ffd640 1 -- 172.21.15.52:0/1213332576 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f74b0052230 con 0x7f74b8034850 2024-03-29T11:25:02.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.040+0000 7f74b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.040+0000 7f74b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:02.027 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:02.027 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:02.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.043+0000 7f74c7594640 10 librados: watch_flush enter 2024-03-29T11:25:02.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.043+0000 7f74c7594640 10 librados: watch_flush exit 2024-03-29T11:25:02.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.043+0000 7f74c7594640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:25:02.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.043+0000 7f74c7594640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.044+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 >> v1:172.21.15.79:6816/407480584 conn(0x7f74a406dc10 legacy=0x7f74a40700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.044+0000 7f74c7594640 10 monclient: shutdown 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.045+0000 7f74c7594640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.045+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 >> v1:172.21.15.52:6790/0 conn(0x7f74b8034850 legacy=0x7f74b8168000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.045+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 shutdown_connections 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.045+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 >> 172.21.15.52:0/1213332576 conn(0x7f74b80058f0 msgr2=0x7f74b80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:02.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.046+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 shutdown_connections 2024-03-29T11:25:02.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.046+0000 7f74c7594640 1 -- 172.21.15.52:0/1213332576 wait complete. 2024-03-29T11:25:02.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:02.046+0000 7f74c7594640 1 librados: shutdown 2024-03-29T11:25:02.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:08.856 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.031+0000 7f8d446ba640 10 monclient: get_monmap_and_config 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.031+0000 7f8d446ba640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.032+0000 7f8d446ba640 10 monclient: monmap: 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:09.032916+0000 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:09.032916+0000 2024-03-29T11:25:09.015 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.033+0000 7f8d446ba640 1 Processor -- start 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.033+0000 7f8d446ba640 1 -- start start 2024-03-29T11:25:09.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.033+0000 7f8d446ba640 10 monclient: init 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c1457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c13f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c0a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0aaef0 con 0x7f8d3c13f650 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c145bd0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c145eb0 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.034+0000 7f8d446ba640 10 monclient(hunting): authenticate will time out at 1239.174586s 2024-03-29T11:25:09.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.039+0000 7f8d41c55640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f8d3c13f650 0x7f8d3c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33919/0 (socket says 172.21.15.52:33919) 2024-03-29T11:25:09.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.039+0000 7f8d41c55640 1 -- 172.21.15.52:0/2047210912 learned_addr learned my addr 172.21.15.52:0/2047210912 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:09.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652115732 0 0) 0x7f8d3c145eb0 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d28003520 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049568277 0 0) 0x7f8d3c0aaef0 con 0x7f8d3c13f650 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:25:09.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.041+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c145eb0 con 0x7f8d3c13f650 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3941469337 0 0) 0x7f8d28003520 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 >> v1:172.21.15.79:6789/0 conn(0x7f8d3c1457f0 legacy=0x7f8d3c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 >> v1:172.21.15.52:6789/0 conn(0x7f8d3c13f650 legacy=0x7f8d3c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 monclient: found mon.noname-b 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c146340 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f8d34001940 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: dump: 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:09.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d34001b60 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 1 -- 172.21.15.52:0/2047210912 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f8d34001ef0 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.042+0000 7f8d41454640 10 monclient: dump: 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:09.028 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:09.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.043+0000 7f8d446ba640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:25:09.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.043+0000 7f8d446ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:09.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.043+0000 7f8d446ba640 4 set_mon_vals no callback set 2024-03-29T11:25:09.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.044+0000 7f8d446ba640 10 monclient: shutdown 2024-03-29T11:25:09.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.044+0000 7f8d446ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.044+0000 7f8d446ba640 1 -- 172.21.15.52:0/2047210912 >> v1:172.21.15.52:6790/0 conn(0x7f8d3c0a9d70 legacy=0x7f8d3c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.044+0000 7f8d446ba640 1 -- 172.21.15.52:0/2047210912 shutdown_connections 2024-03-29T11:25:09.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.044+0000 7f8d446ba640 1 -- 172.21.15.52:0/2047210912 >> 172.21.15.52:0/2047210912 conn(0x7f8d3c066a00 msgr2=0x7f8d3c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.048+0000 7f8d446ba640 1 -- 172.21.15.52:0/2047210912 shutdown_connections 2024-03-29T11:25:09.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.049+0000 7f8d446ba640 1 -- 172.21.15.52:0/2047210912 wait complete. 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.050+0000 7f8d446ba640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.050+0000 7f8d446ba640 10 monclient: monmap: 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:09.033 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.034 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:09.034 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:09.034 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:09.034 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:09.034 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:09.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.051+0000 7f8d446ba640 1 librados: starting msgr at 2024-03-29T11:25:09.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.051+0000 7f8d446ba640 1 librados: starting objecter 2024-03-29T11:25:09.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.051+0000 7f8d446ba640 1 Processor -- start 2024-03-29T11:25:09.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.052+0000 7f8d446ba640 1 -- start start 2024-03-29T11:25:09.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.052+0000 7f8d446ba640 1 librados: setting wanted keys 2024-03-29T11:25:09.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.052+0000 7f8d446ba640 1 librados: calling monclient init 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.052+0000 7f8d446ba640 10 monclient: init 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c0a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c13f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c1457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0ad500 con 0x7f8d3c1457f0 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0ac0b0 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0ad730 con 0x7f8d3c13f650 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.053+0000 7f8d446ba640 10 monclient(hunting): authenticate will time out at 1239.193738s 2024-03-29T11:25:09.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.056+0000 7f8d42c57640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f8d3c1457f0 0x7f8d3c140440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:34087/0 (socket says 172.21.15.52:34087) 2024-03-29T11:25:09.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.056+0000 7f8d42c57640 1 -- 172.21.15.52:0/492452108 learned_addr learned my addr 172.21.15.52:0/492452108 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 client.?.objecter ms_handle_connect 0x7f8d3c1457f0 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 client.?.objecter ms_handle_connect 0x7f8d3c0a9d70 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 client.?.objecter ms_handle_connect 0x7f8d3c13f650 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202494333 0 0) 0x7f8d3c0ad500 con 0x7f8d3c1457f0 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d20004eb0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179764994 0 0) 0x7f8d3c0ac0b0 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.058+0000 7f8d32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c0ad500 con 0x7f8d3c0a9d70 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 476438296 0 0) 0x7f8d20004eb0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3c0ac0b0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f8d340018c0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:09.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.059+0000 7f8d32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3857576873 0 0) 0x7f8d3c0ac0b0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 >> v1:172.21.15.52:6790/0 conn(0x7f8d3c13f650 legacy=0x7f8d3c138190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 >> v1:172.21.15.79:6789/0 conn(0x7f8d3c0a9d70 legacy=0x7f8d3c0ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 monclient: found mon.a 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c142d80 con 0x7f8d3c1457f0 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d34003c20 con 0x7f8d3c1457f0 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f8d34004030 con 0x7f8d3c1457f0 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d32ffd640 10 monclient: dump: 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:09.047 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 10 monclient: _renew_subs 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c0ac710 con 0x7f8d3c1457f0 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 10 monclient: _renew_subs 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c138a80 con 0x7f8d3c1457f0 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.060+0000 7f8d446ba640 1 librados: init done 2024-03-29T11:25:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.061+0000 7f8d33fff640 4 set_mon_vals no callback set 2024-03-29T11:25:09.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.064+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f8d34004190 con 0x7f8d3c1457f0 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.064+0000 7f8d446ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.064+0000 7f8d446ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.064+0000 7f8d446ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.064+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d24005080 con 0x7f8d3c1457f0 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.065+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f8d34084ea0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.065+0000 7f8d32ffd640 10 client.4425.objecter ms_dispatch 0x7f8d3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.065+0000 7f8d42456640 10 monclient: get_auth_request con 0x7f8d20076640 auth_method 0 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.065+0000 7f8d32ffd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.065+0000 7f8d32ffd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.065+0000 7f8d32ffd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:25:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.067+0000 7f8d32ffd640 10 client.4425.objecter ms_handle_connect 0x7f8d20076640 2024-03-29T11:25:09.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.075+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d34050780 con 0x7f8d3c1457f0 2024-03-29T11:25:09.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.075+0000 7f8d32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.075+0000 7f8d32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d446ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d446ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d446ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d24005330 con 0x7f8d3c1457f0 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d32ffd640 1 -- 172.21.15.52:0/492452108 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f8d340526b0 con 0x7f8d3c1457f0 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.410+0000 7f8d32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:09.395 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:09.395 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:09.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.415+0000 7f8d446ba640 10 librados: watch_flush enter 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.415+0000 7f8d446ba640 10 librados: watch_flush exit 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.415+0000 7f8d446ba640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.415+0000 7f8d446ba640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.415+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 >> v1:172.21.15.79:6816/407480584 conn(0x7f8d20076640 legacy=0x7f8d20078b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.415+0000 7f8d446ba640 10 monclient: shutdown 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.416+0000 7f8d446ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.416+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 >> v1:172.21.15.52:6789/0 conn(0x7f8d3c1457f0 legacy=0x7f8d3c140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.416+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 shutdown_connections 2024-03-29T11:25:09.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.416+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 >> 172.21.15.52:0/492452108 conn(0x7f8d3c066a00 msgr2=0x7f8d3c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.417+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 shutdown_connections 2024-03-29T11:25:09.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.417+0000 7f8d446ba640 1 -- 172.21.15.52:0/492452108 wait complete. 2024-03-29T11:25:09.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:09.417+0000 7f8d446ba640 1 librados: shutdown 2024-03-29T11:25:09.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:16.226 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.395+0000 7ff0bd066640 10 monclient: get_monmap_and_config 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.395+0000 7ff0bd066640 10 monclient: build_initial_monmap 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.395+0000 7ff0bd066640 10 monclient: monmap: 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:16.396727+0000 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:16.396727+0000 2024-03-29T11:25:16.379 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:16.380 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:16.380 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:16.380 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:16.380 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:16.380 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:16.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.398+0000 7ff0bd066640 1 Processor -- start 2024-03-29T11:25:16.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.399+0000 7ff0bd066640 1 -- start start 2024-03-29T11:25:16.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.399+0000 7ff0bd066640 10 monclient: init 2024-03-29T11:25:16.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.400+0000 7ff0bd066640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:16.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.400+0000 7ff0bd066640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:16.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.400+0000 7ff0bd066640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b0006e30 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:16.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.400+0000 7ff0bd066640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b0035e00 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:16.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.403+0000 7ff0bd066640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b00c9670 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:16.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.405+0000 7ff0b7fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7ff0b0006e30 0x7ff0b00cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:32789/0 (socket says 172.21.15.52:32789) 2024-03-29T11:25:16.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.405+0000 7ff0b7fff640 1 -- 172.21.15.52:0/3623041661 learned_addr learned my addr 172.21.15.52:0/3623041661 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:16.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.406+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0035970 con 0x7ff0b00c9670 2024-03-29T11:25:16.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.406+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0036280 con 0x7ff0b0035e00 2024-03-29T11:25:16.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.406+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0035640 con 0x7ff0b0006e30 2024-03-29T11:25:16.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.406+0000 7ff0bd066640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:16.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.406+0000 7ff0bd066640 10 monclient(hunting): authenticate will time out at 1246.547034s 2024-03-29T11:25:16.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126553360 0 0) 0x7ff0b0035970 con 0x7ff0b00c9670 2024-03-29T11:25:16.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a8003520 con 0x7ff0b00c9670 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075642407 0 0) 0x7ff0b0035640 con 0x7ff0b0006e30 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b0035970 con 0x7ff0b0006e30 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730706741 0 0) 0x7ff0b0036280 con 0x7ff0b0035e00 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:25:16.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.408+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b0035640 con 0x7ff0b0035e00 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 559243319 0 0) 0x7ff0b0035640 con 0x7ff0b0035e00 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 >> v1:172.21.15.52:6790/0 conn(0x7ff0b0006e30 legacy=0x7ff0b00cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 >> v1:172.21.15.52:6789/0 conn(0x7ff0b00c9670 legacy=0x7ff0b00d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0034e20 con 0x7ff0b0035e00 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:16.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff0a40015b0 con 0x7ff0b0035e00 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0b6ffd640 10 monclient: dump: 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:16.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0bd066640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:25:16.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.409+0000 7ff0bd066640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0a4001800 con 0x7ff0b0035e00 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0b6ffd640 1 -- 172.21.15.52:0/3623041661 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff0a4001b90 con 0x7ff0b0035e00 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0b6ffd640 10 monclient: dump: 2024-03-29T11:25:16.399 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0bd066640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.412+0000 7ff0bd066640 4 set_mon_vals no callback set 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.413+0000 7ff0bd066640 10 monclient: shutdown 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.413+0000 7ff0bd066640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.413+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 >> v1:172.21.15.79:6789/0 conn(0x7ff0b0035e00 legacy=0x7ff0b00c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.414+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 shutdown_connections 2024-03-29T11:25:16.401 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.414+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 >> 172.21.15.52:0/3623041661 conn(0x7ff0b0005c00 msgr2=0x7ff0b0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.401 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.415+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 shutdown_connections 2024-03-29T11:25:16.401 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.415+0000 7ff0bd066640 1 -- 172.21.15.52:0/3623041661 wait complete. 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.416+0000 7ff0bd066640 10 monclient: build_initial_monmap 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.416+0000 7ff0bd066640 10 monclient: monmap: 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:16.402 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.417+0000 7ff0bd066640 1 librados: starting msgr at 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.417+0000 7ff0bd066640 1 librados: starting objecter 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.417+0000 7ff0bd066640 1 Processor -- start 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.417+0000 7ff0bd066640 1 -- start start 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.417+0000 7ff0bd066640 1 librados: setting wanted keys 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.418+0000 7ff0bd066640 1 librados: calling monclient init 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.418+0000 7ff0bd066640 10 monclient: init 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.419+0000 7ff0bd066640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:16.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.419+0000 7ff0bd066640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.419+0000 7ff0bd066640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b0006e30 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b0035e00 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b00c9670 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b00cb8d0 con 0x7ff0b0006e30 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b002deb0 con 0x7ff0b00c9670 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b002e190 con 0x7ff0b0035e00 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:16.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.420+0000 7ff0bd066640 10 monclient(hunting): authenticate will time out at 1246.560903s 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.424+0000 7ff0bc865640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7ff0b00c9670 0x7ff0b016c6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:53971/0 (socket says 172.21.15.52:53971) 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.424+0000 7ff0bc865640 1 -- 172.21.15.52:0/2163501387 learned_addr learned my addr 172.21.15.52:0/2163501387 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 10 client.?.objecter ms_handle_connect 0x7ff0b00c9670 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107032275 0 0) 0x7ff0b002deb0 con 0x7ff0b00c9670 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:25:16.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.425+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09c004eb0 con 0x7ff0b00c9670 2024-03-29T11:25:16.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 client.?.objecter ms_handle_connect 0x7ff0b0035e00 2024-03-29T11:25:16.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572008660 0 0) 0x7ff0b002e190 con 0x7ff0b0035e00 2024-03-29T11:25:16.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b002deb0 con 0x7ff0b0035e00 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 203493193 0 0) 0x7ff0b002deb0 con 0x7ff0b0035e00 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0b002e190 con 0x7ff0b0035e00 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff0a4001b10 con 0x7ff0b0035e00 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.415 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3716697417 0 0) 0x7ff0b002e190 con 0x7ff0b0035e00 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 >> v1:172.21.15.79:6789/0 conn(0x7ff0b00c9670 legacy=0x7ff0b016c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 >> v1:172.21.15.52:6789/0 conn(0x7ff0b0006e30 legacy=0x7ff0b00335e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 monclient: found mon.c 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0033cf0 con 0x7ff0b0035e00 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:16.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.426+0000 7ff0b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 10 monclient: _renew_subs 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0032bd0 con 0x7ff0b0035e00 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 10 monclient: _renew_subs 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b002e5b0 con 0x7ff0b0035e00 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0bd066640 1 librados: init done 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0a40038a0 con 0x7ff0b0035e00 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.427+0000 7ff0b5ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.429+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7ff0a4003cf0 con 0x7ff0b0035e00 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.429+0000 7ff0b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.429+0000 7ff0b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.429+0000 7ff0b4ff9640 10 monclient: dump: 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.429+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7ff0a4003f10 con 0x7ff0b0035e00 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0bd066640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0bd066640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0bd066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:16.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff098005080 con 0x7ff0b0035e00 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7ff0a4083a10 con 0x7ff0b0035e00 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0b4ff9640 10 client.4433.objecter ms_dispatch 0x7ff0b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0b4ff9640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0b4ff9640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.430+0000 7ff0b4ff9640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.433+0000 7ff0b7fff640 10 monclient: get_auth_request con 0x7ff09c06d9f0 auth_method 0 2024-03-29T11:25:16.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.434+0000 7ff0b4ff9640 10 client.4433.objecter ms_handle_connect 0x7ff09c06d9f0 2024-03-29T11:25:16.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.439+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0a404f370 con 0x7ff0b0035e00 2024-03-29T11:25:16.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.439+0000 7ff0b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.439+0000 7ff0b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:16.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.771+0000 7ff0bd066640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.771+0000 7ff0bd066640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.771+0000 7ff0bd066640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:16.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.771+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff098005330 con 0x7ff0b0035e00 2024-03-29T11:25:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.773+0000 7ff0b4ff9640 1 -- 172.21.15.52:0/2163501387 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7ff0a40512a0 con 0x7ff0b0035e00 2024-03-29T11:25:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.773+0000 7ff0b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.773+0000 7ff0b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:16.759 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:16.760 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.776+0000 7ff0bd066640 10 librados: watch_flush enter 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 10 librados: watch_flush exit 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 >> v1:172.21.15.79:6816/407480584 conn(0x7ff09c06d9f0 legacy=0x7ff09c06feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 10 monclient: shutdown 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 >> v1:172.21.15.52:6790/0 conn(0x7ff0b0035e00 legacy=0x7ff0b00c0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 shutdown_connections 2024-03-29T11:25:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.777+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 >> 172.21.15.52:0/2163501387 conn(0x7ff0b0005c00 msgr2=0x7ff0b00c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.778+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 shutdown_connections 2024-03-29T11:25:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.778+0000 7ff0bd066640 1 -- 172.21.15.52:0/2163501387 wait complete. 2024-03-29T11:25:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:16.778+0000 7ff0bd066640 1 librados: shutdown 2024-03-29T11:25:16.779 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:23.586 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.757+0000 7faf6e3b5640 10 monclient: get_monmap_and_config 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.757+0000 7faf6e3b5640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.757+0000 7faf6e3b5640 10 monclient: monmap: 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:23.758796+0000 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:23.758796+0000 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:23.742 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:23.743 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:23.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.758+0000 7faf6e3b5640 1 Processor -- start 2024-03-29T11:25:23.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 1 -- start start 2024-03-29T11:25:23.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient: init 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7faf680b0860 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient(hunting): picked mon.noname-b con 0x7faf6813ef30 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient(hunting): picked mon.noname-c con 0x7faf680af8e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf680b0c40 con 0x7faf680b0860 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf680aed10 con 0x7faf680af8e0 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf680afd50 con 0x7faf6813ef30 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.759+0000 7faf6e3b5640 10 monclient(hunting): authenticate will time out at 1253.899995s 2024-03-29T11:25:23.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.762+0000 7faf677fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7faf6813ef30 0x7faf680af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52035/0 (socket says 172.21.15.52:52035) 2024-03-29T11:25:23.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.762+0000 7faf677fe640 1 -- 172.21.15.52:0/1620758366 learned_addr learned my addr 172.21.15.52:0/1620758366 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:23.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211220682 0 0) 0x7faf680afd50 con 0x7faf6813ef30 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf50003520 con 0x7faf6813ef30 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3214489137 0 0) 0x7faf50003520 con 0x7faf6813ef30 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 >> v1:172.21.15.79:6789/0 conn(0x7faf680af8e0 legacy=0x7faf68149a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 >> v1:172.21.15.52:6789/0 conn(0x7faf680b0860 legacy=0x7faf6813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 monclient: found mon.noname-b 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf680ab1b0 con 0x7faf6813ef30 2024-03-29T11:25:23.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf6e3b5640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf6e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf60004d90 con 0x7faf6813ef30 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf66ffd640 10 monclient: dump: 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.751 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.763+0000 7faf6e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.765+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf60004ff0 con 0x7faf6813ef30 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.765+0000 7faf66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.765+0000 7faf6e3b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:23.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.765+0000 7faf6e3b5640 4 set_mon_vals no callback set 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf66ffd640 1 -- 172.21.15.52:0/1620758366 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf60004d90 con 0x7faf6813ef30 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf66ffd640 10 monclient: dump: 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:23.754 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf6e3b5640 10 monclient: shutdown 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf6e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf6e3b5640 1 -- 172.21.15.52:0/1620758366 >> v1:172.21.15.52:6790/0 conn(0x7faf6813ef30 legacy=0x7faf680af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf6e3b5640 1 -- 172.21.15.52:0/1620758366 shutdown_connections 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.766+0000 7faf6e3b5640 1 -- 172.21.15.52:0/1620758366 >> 172.21.15.52:0/1620758366 conn(0x7faf6806b660 msgr2=0x7faf6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.767+0000 7faf6e3b5640 1 -- 172.21.15.52:0/1620758366 shutdown_connections 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.767+0000 7faf6e3b5640 1 -- 172.21.15.52:0/1620758366 wait complete. 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.767+0000 7faf6e3b5640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.767+0000 7faf6e3b5640 10 monclient: monmap: 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 1 librados: starting msgr at 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 1 librados: starting objecter 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 1 Processor -- start 2024-03-29T11:25:23.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 1 -- start start 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 1 librados: setting wanted keys 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 1 librados: calling monclient init 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.768+0000 7faf6e3b5640 10 monclient: init 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.769+0000 7faf6e3b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.769+0000 7faf6e3b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.769+0000 7faf6e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7faf680af8e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:23.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.769+0000 7faf6e3b5640 10 monclient(hunting): picked mon.noname-b con 0x7faf680b0860 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:23.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.770+0000 7faf67fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7faf680af8e0 0x7faf680b3de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:42558/0 (socket says 172.21.15.52:42558) 2024-03-29T11:25:23.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf6e3b5640 10 monclient(hunting): picked mon.noname-c con 0x7faf6813ef30 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:23.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf6e3b5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf68137260 con 0x7faf680af8e0 2024-03-29T11:25:23.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf6e3b5640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf681375f0 con 0x7faf680b0860 2024-03-29T11:25:23.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf6e3b5640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf68137780 con 0x7faf6813ef30 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf6e3b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf6e3b5640 10 monclient(hunting): authenticate will time out at 1253.912912s 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.772+0000 7faf67fff640 1 -- 172.21.15.52:0/3834702205 learned_addr learned my addr 172.21.15.52:0/3834702205 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 client.?.objecter ms_handle_connect 0x7faf6813ef30 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939863855 0 0) 0x7faf68137780 con 0x7faf6813ef30 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf54004eb0 con 0x7faf6813ef30 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1990415316 0 0) 0x7faf54004eb0 con 0x7faf6813ef30 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf68137780 con 0x7faf6813ef30 2024-03-29T11:25:23.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf580017f0 con 0x7faf6813ef30 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.773+0000 7faf64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.774+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 89165869 0 0) 0x7faf68137780 con 0x7faf6813ef30 2024-03-29T11:25:23.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.774+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 >> v1:172.21.15.79:6789/0 conn(0x7faf680b0860 legacy=0x7faf681e7bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 >> v1:172.21.15.52:6789/0 conn(0x7faf680af8e0 legacy=0x7faf680b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 1 monclient: found mon.c 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf68135580 con 0x7faf6813ef30 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 10 client.?.objecter ms_handle_connect 0x7faf680af8e0 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.775+0000 7faf6e3b5640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 10 monclient: _renew_subs 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:23.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf68135f70 con 0x7faf6813ef30 2024-03-29T11:25:23.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:23.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 10 monclient: _renew_subs 2024-03-29T11:25:23.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:23.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf681e84a0 con 0x7faf6813ef30 2024-03-29T11:25:23.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.776+0000 7faf6e3b5640 1 librados: init done 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.777+0000 7faf4e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.777+0000 7faf4e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.777+0000 7faf4e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.777+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf58001dc0 con 0x7faf6813ef30 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.777+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf48005080 con 0x7faf6813ef30 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.778+0000 7faf64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.778+0000 7faf667fc640 4 set_mon_vals no callback set 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.778+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf58003830 con 0x7faf6813ef30 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.778+0000 7faf64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.778+0000 7faf64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.778+0000 7faf64ff9640 10 monclient: dump: 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.779+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7faf58019880 con 0x7faf6813ef30 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.779+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7faf580838f0 con 0x7faf6813ef30 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.779+0000 7faf64ff9640 10 client.4439.objecter ms_dispatch 0x7faf68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.779+0000 7faf64ff9640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.779+0000 7faf64ff9640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:23.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.779+0000 7faf64ff9640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:25:23.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.780+0000 7faf67fff640 10 monclient: get_auth_request con 0x7faf5406db60 auth_method 0 2024-03-29T11:25:23.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.780+0000 7faf64ff9640 10 client.4439.objecter ms_handle_connect 0x7faf5406db60 2024-03-29T11:25:23.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.786+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf5804f250 con 0x7faf6813ef30 2024-03-29T11:25:23.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.786+0000 7faf64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:23.786+0000 7faf64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:24.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.121+0000 7faf4e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.121+0000 7faf4e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.121+0000 7faf4e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:24.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.121+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf48005330 con 0x7faf6813ef30 2024-03-29T11:25:24.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.122+0000 7faf64ff9640 1 -- 172.21.15.52:0/3834702205 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7faf58051180 con 0x7faf6813ef30 2024-03-29T11:25:24.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.122+0000 7faf64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.122+0000 7faf64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:24.108 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:24.108 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:24.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.127+0000 7faf4e7fc640 10 librados: watch_flush enter 2024-03-29T11:25:24.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 10 librados: watch_flush exit 2024-03-29T11:25:24.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:25:24.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 >> v1:172.21.15.79:6816/407480584 conn(0x7faf5406db60 legacy=0x7faf54070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 10 monclient: shutdown 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 >> v1:172.21.15.52:6790/0 conn(0x7faf6813ef30 legacy=0x7faf681eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 shutdown_connections 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 >> 172.21.15.52:0/3834702205 conn(0x7faf6806b660 msgr2=0x7faf6813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:24.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 shutdown_connections 2024-03-29T11:25:24.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 -- 172.21.15.52:0/3834702205 wait complete. 2024-03-29T11:25:24.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:24.128+0000 7faf4e7fc640 1 librados: shutdown 2024-03-29T11:25:24.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:30.939 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:31.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.102+0000 7f2b1eed9640 10 monclient: get_monmap_and_config 2024-03-29T11:25:31.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.103+0000 7f2b1eed9640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.103+0000 7f2b1eed9640 10 monclient: monmap: 2024-03-29T11:25:31.087 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:31.087 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:31.087 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:31.103950+0000 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:31.103950+0000 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:31.088 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:31.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.104+0000 7f2b1eed9640 1 Processor -- start 2024-03-29T11:25:31.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.104+0000 7f2b1eed9640 1 -- start start 2024-03-29T11:25:31.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.104+0000 7f2b1eed9640 10 monclient: init 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.105+0000 7f2b1eed9640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.105+0000 7f2b1eed9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.105+0000 7f2b1eed9640 10 monclient(hunting): picked mon.noname-a con 0x7f2b181457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 10 monclient(hunting): picked mon.noname-c con 0x7f2b180a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b180aaef0 con 0x7f2b181457f0 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1cc75640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f2b181457f0 0x7f2b1813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:60530/0 (socket says 172.21.15.52:60530) 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1d476640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f2b180a9d70 0x7f2b1814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:45068/0 (socket says 172.21.15.52:45068) 2024-03-29T11:25:31.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1cc75640 1 -- 172.21.15.52:0/1898058529 learned_addr learned my addr 172.21.15.52:0/1898058529 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:31.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b18145bd0 con 0x7f2b180a9d70 2024-03-29T11:25:31.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b18145eb0 con 0x7f2b1813f650 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.106+0000 7f2b1eed9640 10 monclient(hunting): authenticate will time out at 1261.246816s 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217611447 0 0) 0x7f2b180aaef0 con 0x7f2b181457f0 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08003520 con 0x7f2b181457f0 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128705227 0 0) 0x7f2b18145bd0 con 0x7f2b180a9d70 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:25:31.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b180aaef0 con 0x7f2b180a9d70 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376632230 0 0) 0x7f2b18145eb0 con 0x7f2b1813f650 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.107+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b18145bd0 con 0x7f2b1813f650 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2243630980 0 0) 0x7f2b18145bd0 con 0x7f2b1813f650 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 >> v1:172.21.15.79:6789/0 conn(0x7f2b180a9d70 legacy=0x7f2b1814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 >> v1:172.21.15.52:6789/0 conn(0x7f2b181457f0 legacy=0x7f2b1813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:31.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18146340 con 0x7f2b1813f650 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2b040015b0 con 0x7f2b1813f650 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b177fe640 10 monclient: dump: 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.098 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b1eed9640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.108+0000 7f2b1eed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b040017d0 con 0x7f2b1813f650 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b177fe640 1 -- 172.21.15.52:0/1898058529 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2b04001b60 con 0x7f2b1813f650 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b177fe640 10 monclient: dump: 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:31.099 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b1eed9640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.109+0000 7f2b1eed9640 4 set_mon_vals no callback set 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.110+0000 7f2b1eed9640 10 monclient: shutdown 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.111+0000 7f2b1eed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.111+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 >> v1:172.21.15.52:6790/0 conn(0x7f2b1813f650 legacy=0x7f2b180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.111+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 shutdown_connections 2024-03-29T11:25:31.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.111+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 >> 172.21.15.52:0/1898058529 conn(0x7f2b18066a00 msgr2=0x7f2b18066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.112+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 shutdown_connections 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.112+0000 7f2b1eed9640 1 -- 172.21.15.52:0/1898058529 wait complete. 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.113+0000 7f2b1eed9640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.113+0000 7f2b1eed9640 10 monclient: monmap: 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.106 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.114+0000 7f2b1eed9640 1 librados: starting msgr at 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.114+0000 7f2b1eed9640 1 librados: starting objecter 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.114+0000 7f2b1eed9640 1 Processor -- start 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.114+0000 7f2b1eed9640 1 -- start start 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.114+0000 7f2b1eed9640 1 librados: setting wanted keys 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.115+0000 7f2b1eed9640 1 librados: calling monclient init 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.115+0000 7f2b1eed9640 10 monclient: init 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient(hunting): picked mon.noname-c con 0x7f2b180a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1813f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient(hunting): picked mon.noname-a con 0x7f2b181457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b18138d10 con 0x7f2b181457f0 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1813fda0 con 0x7f2b1813f650 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b18141d20 con 0x7f2b180a9d70 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b1eed9640 10 monclient(hunting): authenticate will time out at 1261.256431s 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b17fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f2b1813f650 0x7f2b18140230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:45364/0 (socket says 172.21.15.52:45364) 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b17fff640 1 -- 172.21.15.52:0/2986747024 learned_addr learned my addr 172.21.15.52:0/2986747024 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b157fa640 10 client.?.objecter ms_handle_connect 0x7f2b1813f650 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b157fa640 10 client.?.objecter ms_handle_connect 0x7f2b181457f0 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.116+0000 7f2b157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 client.?.objecter ms_handle_connect 0x7f2b180a9d70 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246628842 0 0) 0x7f2b1813fda0 con 0x7f2b1813f650 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc004eb0 con 0x7f2b1813f650 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689604821 0 0) 0x7f2b18138d10 con 0x7f2b181457f0 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b1813fda0 con 0x7f2b181457f0 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.117+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570365954 0 0) 0x7f2afc004eb0 con 0x7f2b1813f650 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b18138d10 con 0x7f2b1813f650 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816302413 0 0) 0x7f2b18141d20 con 0x7f2b180a9d70 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc004eb0 con 0x7f2b180a9d70 2024-03-29T11:25:31.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2b040015b0 con 0x7f2b1813f650 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): dump: 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:31.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2765698694 0 0) 0x7f2b18138d10 con 0x7f2b1813f650 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 >> v1:172.21.15.52:6790/0 conn(0x7f2b180a9d70 legacy=0x7f2b18142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 >> v1:172.21.15.52:6789/0 conn(0x7f2b181457f0 legacy=0x7f2b181e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 monclient: found mon.b 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18142000 con 0x7f2b1813f650 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b1eed9640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b1eed9640 10 monclient: _renew_subs 2024-03-29T11:25:31.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b1eed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.118+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1813b650 con 0x7f2b1813f650 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b1eed9640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b1eed9640 10 monclient: _renew_subs 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b1eed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b18140b20 con 0x7f2b1813f650 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b1eed9640 1 librados: init done 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b040038a0 con 0x7f2b1813f650 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2b04003bf0 con 0x7f2b1813f650 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:31.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b157fa640 10 monclient: dump: 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.119+0000 7f2b16ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:31.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.121+0000 7f2b1eed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.121+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f2b04003d50 con 0x7f2b1813f650 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.122+0000 7f2b1eed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.122+0000 7f2b1eed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.122+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af8005080 con 0x7f2b1813f650 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.123+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f2b04084b30 con 0x7f2b1813f650 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.123+0000 7f2b157fa640 10 client.4402.objecter ms_dispatch 0x7f2b18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.123+0000 7f2b157fa640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.123+0000 7f2b157fa640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.123+0000 7f2b157fa640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.123+0000 7f2b1cc75640 10 monclient: get_auth_request con 0x7f2afc06dea0 auth_method 0 2024-03-29T11:25:31.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.124+0000 7f2b157fa640 10 client.4402.objecter ms_handle_connect 0x7f2afc06dea0 2024-03-29T11:25:31.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.129+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b04050410 con 0x7f2b1813f650 2024-03-29T11:25:31.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.130+0000 7f2b157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.130+0000 7f2b157fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:31.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.522+0000 7f2b1eed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.522+0000 7f2b1eed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.522+0000 7f2b1eed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:25:31.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.522+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2af8005330 con 0x7f2b1813f650 2024-03-29T11:25:31.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.523+0000 7f2b157fa640 1 -- 172.21.15.52:0/2986747024 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f2b04052340 con 0x7f2b1813f650 2024-03-29T11:25:31.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.524+0000 7f2b157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.524+0000 7f2b157fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:31.508 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:31.508 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:31.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.526+0000 7f2b1eed9640 10 librados: watch_flush enter 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 10 librados: watch_flush exit 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 >> v1:172.21.15.79:6816/407480584 conn(0x7f2afc06dea0 legacy=0x7f2afc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 10 monclient: shutdown 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 >> v1:172.21.15.79:6789/0 conn(0x7f2b1813f650 legacy=0x7f2b18140230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 shutdown_connections 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 >> 172.21.15.52:0/2986747024 conn(0x7f2b18066a00 msgr2=0x7f2b180b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 shutdown_connections 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 -- 172.21.15.52:0/2986747024 wait complete. 2024-03-29T11:25:31.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:31.527+0000 7f2b1eed9640 1 librados: shutdown 2024-03-29T11:25:31.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:38.333 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.482+0000 7f381169a640 10 monclient: get_monmap_and_config 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.482+0000 7f381169a640 10 monclient: build_initial_monmap 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.482+0000 7f381169a640 10 monclient: monmap: 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:38.483523+0000 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:38.483523+0000 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:38.467 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:38.468 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:38.468 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:38.468 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:38.468 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.483+0000 7f381169a640 1 Processor -- start 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.483+0000 7f381169a640 1 -- start start 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.483+0000 7f381169a640 10 monclient: init 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.484+0000 7f381169a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.484+0000 7f381169a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.484+0000 7f381169a640 10 monclient(hunting): picked mon.noname-c con 0x7f380c1457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.484+0000 7f381169a640 10 monclient(hunting): picked mon.noname-b con 0x7f380c13f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.484+0000 7f380a7fc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f380c13f650 0x7f380c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:59456/0 (socket says 172.21.15.52:59456) 2024-03-29T11:25:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.484+0000 7f380affd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f380c1457f0 0x7f380c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:53210/0 (socket says 172.21.15.52:53210) 2024-03-29T11:25:38.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f381169a640 10 monclient(hunting): picked mon.noname-a con 0x7f380c0a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:38.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c0aaef0 con 0x7f380c0a9d70 2024-03-29T11:25:38.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c145bd0 con 0x7f380c1457f0 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c145eb0 con 0x7f380c13f650 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f381169a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f381169a640 10 monclient(hunting): authenticate will time out at 1268.626029s 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.485+0000 7f380a7fc640 1 -- 172.21.15.52:0/1205378159 learned_addr learned my addr 172.21.15.52:0/1205378159 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043751380 0 0) 0x7f380c145eb0 con 0x7f380c13f650 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f4003520 con 0x7f380c13f650 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1640497590 0 0) 0x7f37f4003520 con 0x7f380c13f650 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 >> v1:172.21.15.79:6789/0 conn(0x7f380c1457f0 legacy=0x7f380c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:38.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.486+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 >> v1:172.21.15.52:6789/0 conn(0x7f380c0a9d70 legacy=0x7f380c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:38.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 1 monclient: found mon.noname-b 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c146340 con 0x7f380c13f650 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3800001690 con 0x7f380c13f650 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f3809ffb640 10 monclient: dump: 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f381169a640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:25:38.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.489+0000 7f381169a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38000018f0 con 0x7f380c13f650 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f3809ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f3809ffb640 1 -- 172.21.15.52:0/1205378159 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3800001d40 con 0x7f380c13f650 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f3809ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f3809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f3809ffb640 10 monclient: dump: 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:38.478 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f381169a640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.490+0000 7f381169a640 4 set_mon_vals no callback set 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 10 monclient: shutdown 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 >> v1:172.21.15.52:6790/0 conn(0x7f380c13f650 legacy=0x7f380c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 shutdown_connections 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 >> 172.21.15.52:0/1205378159 conn(0x7f380c066a00 msgr2=0x7f380c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 shutdown_connections 2024-03-29T11:25:38.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.491+0000 7f381169a640 1 -- 172.21.15.52:0/1205378159 wait complete. 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.492+0000 7f381169a640 10 monclient: build_initial_monmap 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.492+0000 7f381169a640 10 monclient: monmap: 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:38.484 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.492+0000 7f381169a640 1 librados: starting msgr at 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.492+0000 7f381169a640 1 librados: starting objecter 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.493+0000 7f381169a640 1 Processor -- start 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.493+0000 7f381169a640 1 -- start start 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.493+0000 7f381169a640 1 librados: setting wanted keys 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.493+0000 7f381169a640 1 librados: calling monclient init 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.493+0000 7f381169a640 10 monclient: init 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient(hunting): picked mon.noname-a con 0x7f380c0a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient(hunting): picked mon.noname-b con 0x7f380c13f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient(hunting): picked mon.noname-c con 0x7f380c1457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f380b7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f380c1457f0 0x7f380c1398a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:33008/0 (socket says 172.21.15.52:33008) 2024-03-29T11:25:38.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f380b7fe640 1 -- 172.21.15.52:0/2529326842 learned_addr learned my addr 172.21.15.52:0/2529326842 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c072030 con 0x7f380c0a9d70 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c139fb0 con 0x7f380c13f650 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 10 client.?.objecter ms_handle_connect 0x7f380c1457f0 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c0b1010 con 0x7f380c1457f0 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f381169a640 10 monclient(hunting): authenticate will time out at 1268.634570s 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34206523 0 0) 0x7f380c0b1010 con 0x7f380c1457f0 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f8004eb0 con 0x7f380c1457f0 2024-03-29T11:25:38.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1696674290 0 0) 0x7f37f8004eb0 con 0x7f380c1457f0 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.494+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f380c0b1010 con 0x7f380c1457f0 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3804004380 con 0x7f380c1457f0 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 10 monclient(hunting): dump: 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:38.487 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3506534822 0 0) 0x7f380c0b1010 con 0x7f380c1457f0 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 >> v1:172.21.15.79:6789/0 conn(0x7f380c13f650 legacy=0x7f380c139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.495+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 >> v1:172.21.15.52:6789/0 conn(0x7f380c0a9d70 legacy=0x7f380c06fd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 1 monclient: found mon.c 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c071b50 con 0x7f380c1457f0 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f37f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f381169a640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:25:38.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f381169a640 10 monclient: _renew_subs 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f381169a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.496+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c070470 con 0x7f380c1457f0 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38040044e0 con 0x7f380c1457f0 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f37f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f38040014f0 con 0x7f380c1457f0 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f37f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f37f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f37f37fe640 10 monclient: dump: 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:38.489 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f381169a640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f381169a640 10 monclient: _renew_subs 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f381169a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c0b1510 con 0x7f380c1457f0 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f381169a640 1 librados: init done 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.497+0000 7f38097fa640 4 set_mon_vals no callback set 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.498+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f3804001650 con 0x7f380c1457f0 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f381169a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f381169a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f381169a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c075f10 con 0x7f380c1457f0 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f3804084da0 con 0x7f380c1457f0 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f37f37fe640 10 client.4451.objecter ms_dispatch 0x7f380c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f37f37fe640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f37f37fe640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f37f37fe640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.499+0000 7f380affd640 10 monclient: get_auth_request con 0x7f37f806db30 auth_method 0 2024-03-29T11:25:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.500+0000 7f37f37fe640 10 client.4451.objecter ms_handle_connect 0x7f37f806db30 2024-03-29T11:25:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.507+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38040506e0 con 0x7f380c1457f0 2024-03-29T11:25:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.507+0000 7f37f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.507+0000 7f37f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.822+0000 7f381169a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.822+0000 7f381169a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.822+0000 7f381169a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.822+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f380c0b3680 con 0x7f380c1457f0 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.823+0000 7f37f37fe640 1 -- 172.21.15.52:0/2529326842 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f3804052610 con 0x7f380c1457f0 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.823+0000 7f37f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.823+0000 7f37f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:38.808 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:38.808 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 10 librados: watch_flush enter 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 10 librados: watch_flush exit 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 >> v1:172.21.15.79:6816/407480584 conn(0x7f37f806db30 legacy=0x7f37f806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 10 monclient: shutdown 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 >> v1:172.21.15.52:6790/0 conn(0x7f380c1457f0 legacy=0x7f380c1398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 shutdown_connections 2024-03-29T11:25:38.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 >> 172.21.15.52:0/2529326842 conn(0x7f380c066a00 msgr2=0x7f380c13a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 shutdown_connections 2024-03-29T11:25:38.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 -- 172.21.15.52:0/2529326842 wait complete. 2024-03-29T11:25:38.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:38.826+0000 7f381169a640 1 librados: shutdown 2024-03-29T11:25:38.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:45.633 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:45.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.792+0000 7f30f897a640 10 monclient: get_monmap_and_config 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.792+0000 7f30f897a640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.792+0000 7f30f897a640 10 monclient: monmap: 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:45.793273+0000 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:45.793273+0000 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.777 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:45.778 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:45.778 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:45.778 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.793+0000 7f30f897a640 1 Processor -- start 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.793+0000 7f30f897a640 1 -- start start 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.793+0000 7f30f897a640 10 monclient: init 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient(hunting): picked mon.noname-c con 0x7f30f41457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient(hunting): picked mon.noname-b con 0x7f30f413f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient(hunting): picked mon.noname-a con 0x7f30f40a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40aaef0 con 0x7f30f40a9d70 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f4145bd0 con 0x7f30f41457f0 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f4145eb0 con 0x7f30f413f650 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f897a640 10 monclient(hunting): authenticate will time out at 1275.934371s 2024-03-29T11:25:45.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f259c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f30f41457f0 0x7f30f413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:46956/0 (socket says 172.21.15.52:46956) 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.794+0000 7f30f259c640 1 -- 172.21.15.52:0/3649842435 learned_addr learned my addr 172.21.15.52:0/3649842435 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159967969 0 0) 0x7f30f4145eb0 con 0x7f30f413f650 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc003520 con 0x7f30f413f650 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 690814914 0 0) 0x7f30dc003520 con 0x7f30f413f650 2024-03-29T11:25:45.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 >> v1:172.21.15.79:6789/0 conn(0x7f30f41457f0 legacy=0x7f30f413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.795+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 >> v1:172.21.15.52:6789/0 conn(0x7f30f40a9d70 legacy=0x7f30f414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:45.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 1 monclient: found mon.noname-b 2024-03-29T11:25:45.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4146340 con 0x7f30f413f650 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30e4001690 con 0x7f30f413f650 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: dump: 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.782 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.783 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:45.783 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:45.783 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:45.783 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f897a640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:25:45.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f897a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e4001920 con 0x7f30f413f650 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f897a640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.797+0000 7f30f897a640 4 set_mon_vals no callback set 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f159a640 1 -- 172.21.15.52:0/3649842435 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30e4001d30 con 0x7f30f413f650 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f159a640 10 monclient: dump: 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:45.784 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f897a640 10 monclient: shutdown 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f897a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.798+0000 7f30f897a640 1 -- 172.21.15.52:0/3649842435 >> v1:172.21.15.52:6790/0 conn(0x7f30f413f650 legacy=0x7f30f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.799+0000 7f30f897a640 1 -- 172.21.15.52:0/3649842435 shutdown_connections 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.799+0000 7f30f897a640 1 -- 172.21.15.52:0/3649842435 >> 172.21.15.52:0/3649842435 conn(0x7f30f4066a00 msgr2=0x7f30f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.799+0000 7f30f897a640 1 -- 172.21.15.52:0/3649842435 shutdown_connections 2024-03-29T11:25:45.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.799+0000 7f30f897a640 1 -- 172.21.15.52:0/3649842435 wait complete. 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.800+0000 7f30f897a640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.800+0000 7f30f897a640 10 monclient: monmap: 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.787 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 1 librados: starting msgr at 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 1 librados: starting objecter 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 1 Processor -- start 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 1 -- start start 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 1 librados: setting wanted keys 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 1 librados: calling monclient init 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.801+0000 7f30f897a640 10 monclient: init 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient(hunting): picked mon.noname-a con 0x7f30f40a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:45.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient(hunting): picked mon.noname-b con 0x7f30f413f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient(hunting): picked mon.noname-c con 0x7f30f41457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f4138930 con 0x7f30f40a9d70 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40ac6e0 con 0x7f30f413f650 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40ad600 con 0x7f30f41457f0 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f897a640 10 monclient(hunting): authenticate will time out at 1275.942322s 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f1d9b640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f30f413f650 0x7f30f4137b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:47134/0 (socket says 172.21.15.52:47134) 2024-03-29T11:25:45.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30f1d9b640 1 -- 172.21.15.52:0/1653653385 learned_addr learned my addr 172.21.15.52:0/1653653385 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:45.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30e2ffd640 10 client.?.objecter ms_handle_connect 0x7f30f413f650 2024-03-29T11:25:45.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.802+0000 7f30e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 client.?.objecter ms_handle_connect 0x7f30f41457f0 2024-03-29T11:25:45.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214682970 0 0) 0x7f30f40ac6e0 con 0x7f30f413f650 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4004eb0 con 0x7f30f413f650 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245443298 0 0) 0x7f30f40ad600 con 0x7f30f41457f0 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.803+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f40ac6e0 con 0x7f30f41457f0 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073644885 0 0) 0x7f30f40ac6e0 con 0x7f30f41457f0 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30f40ad600 con 0x7f30f41457f0 2024-03-29T11:25:45.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30e80017b0 con 0x7f30f41457f0 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:45.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151295925 0 0) 0x7f30d4004eb0 con 0x7f30f413f650 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d4005990 con 0x7f30f413f650 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30e4001650 con 0x7f30f413f650 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:45.793 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1865416064 0 0) 0x7f30f40ad600 con 0x7f30f41457f0 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 >> v1:172.21.15.79:6789/0 conn(0x7f30f413f650 legacy=0x7f30f4137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 >> v1:172.21.15.52:6789/0 conn(0x7f30f40a9d70 legacy=0x7f30f4140320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.804+0000 7f30e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 1 monclient: found mon.c 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f40ad8e0 con 0x7f30f41457f0 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:25:45.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 10 monclient: _renew_subs 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f4140a30 con 0x7f30f41457f0 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 10 monclient: _renew_subs 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f4142ca0 con 0x7f30f41457f0 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30f897a640 1 librados: init done 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e8001cc0 con 0x7f30f41457f0 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f30e80017b0 con 0x7f30f41457f0 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.805+0000 7f30e2ffd640 10 monclient: dump: 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:45.795 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:45.796 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.806+0000 7f30f0d99640 4 set_mon_vals no callback set 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.806+0000 7f30f897a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.807+0000 7f30f897a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.807+0000 7f30f897a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.807+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d8005080 con 0x7f30f41457f0 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.807+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f30e8019460 con 0x7f30f41457f0 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30f259c640 10 monclient: get_auth_request con 0x7f30d406e2b0 auth_method 0 2024-03-29T11:25:45.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f30e80532a0 con 0x7f30f41457f0 2024-03-29T11:25:45.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30e2ffd640 10 client.4457.objecter ms_dispatch 0x7f30f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:45.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30e2ffd640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:45.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30e2ffd640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:45.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30e2ffd640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:25:45.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.808+0000 7f30e2ffd640 10 client.4457.objecter ms_handle_connect 0x7f30d406e2b0 2024-03-29T11:25:45.800 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.816+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30e804f540 con 0x7f30f41457f0 2024-03-29T11:25:45.800 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.816+0000 7f30e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.800 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:45.816+0000 7f30e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:46.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30f897a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30f897a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30f897a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:25:46.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30d8005330 con 0x7f30f41457f0 2024-03-29T11:25:46.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30e2ffd640 1 -- 172.21.15.52:0/1653653385 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f30e804f6e0 con 0x7f30f41457f0 2024-03-29T11:25:46.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.130+0000 7f30e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:46.116 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:46.116 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.133+0000 7f30f897a640 10 librados: watch_flush enter 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.133+0000 7f30f897a640 10 librados: watch_flush exit 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.133+0000 7f30f897a640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.133+0000 7f30f897a640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 >> v1:172.21.15.79:6816/407480584 conn(0x7f30d406e2b0 legacy=0x7f30d4070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 10 monclient: shutdown 2024-03-29T11:25:46.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:46.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 >> v1:172.21.15.52:6790/0 conn(0x7f30f41457f0 legacy=0x7f30f4138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 shutdown_connections 2024-03-29T11:25:46.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 >> 172.21.15.52:0/1653653385 conn(0x7f30f4066a00 msgr2=0x7f30f40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:46.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 shutdown_connections 2024-03-29T11:25:46.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 -- 172.21.15.52:0/1653653385 wait complete. 2024-03-29T11:25:46.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:46.134+0000 7f30f897a640 1 librados: shutdown 2024-03-29T11:25:46.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:52.940 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.109+0000 7fb553c0f640 10 monclient: get_monmap_and_config 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.109+0000 7fb553c0f640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.109+0000 7fb553c0f640 10 monclient: monmap: 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:25:53.110284+0000 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:25:53.110284+0000 2024-03-29T11:25:53.094 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.095 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:53.095 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:53.095 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:25:53.095 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:25:53.095 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:53.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.110+0000 7fb553c0f640 1 Processor -- start 2024-03-29T11:25:53.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.112+0000 7fb553c0f640 1 -- start start 2024-03-29T11:25:53.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.112+0000 7fb553c0f640 10 monclient: init 2024-03-29T11:25:53.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fb54c1422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb54c142690 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fb54c0b17d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c142c80 con 0x7fb54c0b17d0 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c0aae40 con 0x7fb54c1422b0 2024-03-29T11:25:53.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c13fbd0 con 0x7fb54c142690 2024-03-29T11:25:53.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.113+0000 7fb553c0f640 10 monclient(hunting): authenticate will time out at 1283.253338s 2024-03-29T11:25:53.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.116+0000 7fb5511aa640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb54c142690 0x7fb54c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37207/0 (socket says 172.21.15.52:37207) 2024-03-29T11:25:53.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.116+0000 7fb5511aa640 1 -- 172.21.15.52:0/2151999755 learned_addr learned my addr 172.21.15.52:0/2151999755 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039679524 0 0) 0x7fb54c13fbd0 con 0x7fb54c142690 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb538003520 con 0x7fb54c142690 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602678108 0 0) 0x7fb538003520 con 0x7fb54c142690 2024-03-29T11:25:53.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 >> v1:172.21.15.79:6789/0 conn(0x7fb54c1422b0 legacy=0x7fb54c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 >> v1:172.21.15.52:6789/0 conn(0x7fb54c0b17d0 legacy=0x7fb54c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 monclient: found mon.noname-b 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c151940 con 0x7fb54c142690 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb544001a10 con 0x7fb54c142690 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: dump: 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:53.104 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb553c0f640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb553c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb544001b70 con 0x7fb54c142690 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 1 -- 172.21.15.52:0/2151999755 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb544001fc0 con 0x7fb54c142690 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.117+0000 7fb5509a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.118+0000 7fb5509a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.118+0000 7fb5509a9640 10 monclient: dump: 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:53.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.118+0000 7fb553c0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:53.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.118+0000 7fb553c0f640 4 set_mon_vals no callback set 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.119+0000 7fb553c0f640 10 monclient: shutdown 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.119+0000 7fb553c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.119+0000 7fb553c0f640 1 -- 172.21.15.52:0/2151999755 >> v1:172.21.15.52:6790/0 conn(0x7fb54c142690 legacy=0x7fb54c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.121+0000 7fb553c0f640 1 -- 172.21.15.52:0/2151999755 shutdown_connections 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.121+0000 7fb553c0f640 1 -- 172.21.15.52:0/2151999755 >> 172.21.15.52:0/2151999755 conn(0x7fb54c066a00 msgr2=0x7fb54c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.122+0000 7fb553c0f640 1 -- 172.21.15.52:0/2151999755 shutdown_connections 2024-03-29T11:25:53.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.122+0000 7fb553c0f640 1 -- 172.21.15.52:0/2151999755 wait complete. 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.123+0000 7fb553c0f640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.123+0000 7fb553c0f640 10 monclient: monmap: 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:53.110 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:25:53.111 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:25:53.111 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:25:53.111 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:53.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 1 librados: starting msgr at 2024-03-29T11:25:53.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 1 librados: starting objecter 2024-03-29T11:25:53.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 1 Processor -- start 2024-03-29T11:25:53.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 1 -- start start 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 1 librados: setting wanted keys 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 1 librados: calling monclient init 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.124+0000 7fb553c0f640 10 monclient: init 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.125+0000 7fb553c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.125+0000 7fb553c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.125+0000 7fb553c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fb54c0b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.125+0000 7fb553c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fb54c1422b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.126+0000 7fb553c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb54c142690 addr v1:172.21.15.79:6789/0 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.126+0000 7fb553c0f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c12f680 con 0x7fb54c1422b0 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.126+0000 7fb553c0f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c06ed60 con 0x7fb54c142690 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.126+0000 7fb553c0f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c06f040 con 0x7fb54c0b17d0 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.126+0000 7fb553c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.127+0000 7fb553c0f640 10 monclient(hunting): authenticate will time out at 1283.267129s 2024-03-29T11:25:53.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.130+0000 7fb5511aa640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb54c1422b0 0x7fb54c12e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:53087/0 (socket says 172.21.15.52:53087) 2024-03-29T11:25:53.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.130+0000 7fb5511aa640 1 -- 172.21.15.52:0/2266331035 learned_addr learned my addr 172.21.15.52:0/2266331035 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 client.?.objecter ms_handle_connect 0x7fb54c1422b0 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065557023 0 0) 0x7fb54c12f680 con 0x7fb54c1422b0 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:25:53.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb530004eb0 con 0x7fb54c1422b0 2024-03-29T11:25:53.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 client.?.objecter ms_handle_connect 0x7fb54c0b17d0 2024-03-29T11:25:53.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1442259740 0 0) 0x7fb530004eb0 con 0x7fb54c1422b0 2024-03-29T11:25:53.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb54c12f680 con 0x7fb54c1422b0 2024-03-29T11:25:53.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb544001f80 con 0x7fb54c1422b0 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): dump: 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.120 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:53.121 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:53.121 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:53.121 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:53.121 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:53.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.132+0000 7fb5427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 10 client.?.objecter ms_handle_connect 0x7fb54c142690 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 356349414 0 0) 0x7fb54c12f680 con 0x7fb54c1422b0 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 >> v1:172.21.15.52:6790/0 conn(0x7fb54c0b17d0 legacy=0x7fb54c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 >> v1:172.21.15.79:6789/0 conn(0x7fb54c142690 legacy=0x7fb54c12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 1 monclient: found mon.a 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c1314e0 con 0x7fb54c1422b0 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.135+0000 7fb5427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 10 monclient: _renew_subs 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c06de90 con 0x7fb54c1422b0 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 10 monclient: _renew_subs 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c132360 con 0x7fb54c1422b0 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.136+0000 7fb553c0f640 1 librados: init done 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb553c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb553c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb553c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:53.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c075f10 con 0x7fb54c1422b0 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb544004e10 con 0x7fb54c1422b0 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb5440020e0 con 0x7fb54c1422b0 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5427fc640 10 monclient: dump: 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:25:53.127 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.138+0000 7fb5437fe640 4 set_mon_vals no callback set 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fb54401afd0 con 0x7fb54c1422b0 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fb544054eb0 con 0x7fb54c1422b0 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5427fc640 10 client.4440.objecter ms_dispatch 0x7fb54c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5427fc640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:53.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5427fc640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:53.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5519ab640 10 monclient: get_auth_request con 0x7fb53006db70 auth_method 0 2024-03-29T11:25:53.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.139+0000 7fb5427fc640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T11:25:53.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.140+0000 7fb5427fc640 10 client.4440.objecter ms_handle_connect 0x7fb53006db70 2024-03-29T11:25:53.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.148+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5440509c0 con 0x7fb54c1422b0 2024-03-29T11:25:53.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.148+0000 7fb5427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.148+0000 7fb5427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:53.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.477+0000 7fb553c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.477+0000 7fb553c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.477+0000 7fb553c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:25:53.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.477+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb54c074f10 con 0x7fb54c1422b0 2024-03-29T11:25:53.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.481+0000 7fb5427fc640 1 -- 172.21.15.52:0/2266331035 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fb5440509c0 con 0x7fb54c1422b0 2024-03-29T11:25:53.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.481+0000 7fb5427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.481+0000 7fb5427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:53.467 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:25:53.467 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:53.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 10 librados: watch_flush enter 2024-03-29T11:25:53.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 10 librados: watch_flush exit 2024-03-29T11:25:53.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 10 client.4440.objecter successfully canceled tick 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 >> v1:172.21.15.79:6816/407480584 conn(0x7fb53006db70 legacy=0x7fb530070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 10 monclient: shutdown 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 >> v1:172.21.15.52:6789/0 conn(0x7fb54c1422b0 legacy=0x7fb54c12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 shutdown_connections 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 >> 172.21.15.52:0/2266331035 conn(0x7fb54c066a00 msgr2=0x7fb54c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 shutdown_connections 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 -- 172.21.15.52:0/2266331035 wait complete. 2024-03-29T11:25:53.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:25:53.487+0000 7fb553c0f640 1 librados: shutdown 2024-03-29T11:25:53.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:00.301 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.465+0000 7f389dc8d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.465+0000 7f389dc8d640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.465+0000 7f389dc8d640 10 monclient: monmap: 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:00.466626+0000 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:00.466626+0000 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:00.452 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:00.453 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.467+0000 7f389dc8d640 1 Processor -- start 2024-03-29T11:26:00.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.467+0000 7f389dc8d640 1 -- start start 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.467+0000 7f389dc8d640 10 monclient: init 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient(hunting): picked mon.noname-c con 0x7f38980a9930 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient(hunting): picked mon.noname-b con 0x7f3898149dc0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient(hunting): picked mon.noname-a con 0x7f389814ff40 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980aae60 con 0x7f389814ff40 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980b4310 con 0x7f38980a9930 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980afdd0 con 0x7f3898149dc0 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.468+0000 7f389dc8d640 10 monclient(hunting): authenticate will time out at 1290.608281s 2024-03-29T11:26:00.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.474+0000 7f38977fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f38980a9930 0x7f3898143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:50827/0 (socket says 172.21.15.52:50827) 2024-03-29T11:26:00.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.474+0000 7f38977fe640 1 -- 172.21.15.52:0/172919877 learned_addr learned my addr 172.21.15.52:0/172919877 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:00.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511411069 0 0) 0x7f38980b4310 con 0x7f38980a9930 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3880003520 con 0x7f38980a9930 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230887358 0 0) 0x7f38980afdd0 con 0x7f3898149dc0 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:26:00.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38980b4310 con 0x7f3898149dc0 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.475+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437079853 0 0) 0x7f38980aae60 con 0x7f389814ff40 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38980afdd0 con 0x7f389814ff40 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761149537 0 0) 0x7f38980b4310 con 0x7f3898149dc0 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 >> v1:172.21.15.79:6789/0 conn(0x7f38980a9930 legacy=0x7f3898143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 >> v1:172.21.15.52:6789/0 conn(0x7f389814ff40 legacy=0x7f3898152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389814a950 con 0x7f3898149dc0 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f388c001520 con 0x7f3898149dc0 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: dump: 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.463 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f388c001740 con 0x7f3898149dc0 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 1 -- 172.21.15.52:0/172919877 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f388c001ad0 con 0x7f3898149dc0 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f38967fc640 10 monclient: dump: 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.464 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f389dc8d640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f389dc8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:00.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.476+0000 7f389dc8d640 4 set_mon_vals no callback set 2024-03-29T11:26:00.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.477+0000 7f389dc8d640 10 monclient: shutdown 2024-03-29T11:26:00.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.477+0000 7f389dc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.477+0000 7f389dc8d640 1 -- 172.21.15.52:0/172919877 >> v1:172.21.15.52:6790/0 conn(0x7f3898149dc0 legacy=0x7f389814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.479+0000 7f389dc8d640 1 -- 172.21.15.52:0/172919877 shutdown_connections 2024-03-29T11:26:00.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.479+0000 7f389dc8d640 1 -- 172.21.15.52:0/172919877 >> 172.21.15.52:0/172919877 conn(0x7f38980b35e0 msgr2=0x7f38980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.479+0000 7f389dc8d640 1 -- 172.21.15.52:0/172919877 shutdown_connections 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.479+0000 7f389dc8d640 1 -- 172.21.15.52:0/172919877 wait complete. 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 10 monclient: monmap: 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.470 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 1 librados: starting msgr at 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 1 librados: starting objecter 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 1 Processor -- start 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 1 -- start start 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 1 librados: setting wanted keys 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 1 librados: calling monclient init 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.480+0000 7f389dc8d640 10 monclient: init 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient(hunting): picked mon.noname-c con 0x7f38980a9930 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:00.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient(hunting): picked mon.noname-a con 0x7f3898149dc0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:00.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient(hunting): picked mon.noname-b con 0x7f389814ff40 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:00.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980723a0 con 0x7f3898149dc0 2024-03-29T11:26:00.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389808bb30 con 0x7f389814ff40 2024-03-29T11:26:00.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3898072620 con 0x7f38980a9930 2024-03-29T11:26:00.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.481+0000 7f389dc8d640 10 monclient(hunting): authenticate will time out at 1290.621676s 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.488+0000 7f38977fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f38980a9930 0x7f389813fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:34951/0 (socket says 172.21.15.52:34951) 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.488+0000 7f38977fe640 1 -- 172.21.15.52:0/231272293 learned_addr learned my addr 172.21.15.52:0/231272293 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.491+0000 7f387ffff640 10 client.?.objecter ms_handle_connect 0x7f38980a9930 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.491+0000 7f387ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 client.?.objecter ms_handle_connect 0x7f3898149dc0 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653874112 0 0) 0x7f3898072620 con 0x7f38980a9930 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3884004eb0 con 0x7f38980a9930 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965623718 0 0) 0x7f38980723a0 con 0x7f3898149dc0 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.492+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3898072620 con 0x7f3898149dc0 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2153956284 0 0) 0x7f3884004eb0 con 0x7f38980a9930 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38980723a0 con 0x7f38980a9930 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3890001c90 con 0x7f38980a9930 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:00.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): dump: 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893264452 0 0) 0x7f3898072620 con 0x7f3898149dc0 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3884004eb0 con 0x7f3898149dc0 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f388c001460 con 0x7f3898149dc0 2024-03-29T11:26:00.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 10 monclient(hunting): dump: 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2553479274 0 0) 0x7f3884004eb0 con 0x7f3898149dc0 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.493+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 >> v1:172.21.15.52:6790/0 conn(0x7f38980a9930 legacy=0x7f389813fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 >> v1:172.21.15.79:6789/0 conn(0x7f389814ff40 legacy=0x7f3898137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 1 monclient: found mon.a 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389806f300 con 0x7f3898149dc0 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 10 monclient: _renew_subs 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389806f5e0 con 0x7f3898149dc0 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 10 monclient: _renew_subs 2024-03-29T11:26:00.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898138a20 con 0x7f3898149dc0 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f389dc8d640 1 librados: init done 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f388c0015c0 con 0x7f3898149dc0 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f388c003a40 con 0x7f3898149dc0 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.494+0000 7f387ffff640 10 monclient: dump: 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.487 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.495+0000 7f3895ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.496+0000 7f389dc8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.496+0000 7f389dc8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.496+0000 7f389dc8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.496+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3878005080 con 0x7f3898149dc0 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.497+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f388c019a10 con 0x7f3898149dc0 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f38977fe640 10 monclient: get_auth_request con 0x7f388406dec0 auth_method 0 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f388c083800 con 0x7f3898149dc0 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f387ffff640 10 client.4446.objecter ms_dispatch 0x7f3898000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f387ffff640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f387ffff640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f387ffff640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T11:26:00.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.498+0000 7f387ffff640 10 client.4446.objecter ms_handle_connect 0x7f388406dec0 2024-03-29T11:26:00.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.505+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f388c04ef10 con 0x7f3898149dc0 2024-03-29T11:26:00.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.505+0000 7f387ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.505+0000 7f387ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:00.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.839+0000 7f389dc8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.839+0000 7f389dc8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.839+0000 7f389dc8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:00.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.839+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3878005330 con 0x7f3898149dc0 2024-03-29T11:26:00.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.841+0000 7f387ffff640 1 -- 172.21.15.52:0/231272293 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f388c051010 con 0x7f3898149dc0 2024-03-29T11:26:00.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.841+0000 7f387ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.841+0000 7f387ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:00.827 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:00.827 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:00.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 10 librados: watch_flush enter 2024-03-29T11:26:00.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 10 librados: watch_flush exit 2024-03-29T11:26:00.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T11:26:00.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 10 client.4446.objecter successfully canceled tick 2024-03-29T11:26:00.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 >> v1:172.21.15.79:6816/407480584 conn(0x7f388406dec0 legacy=0x7f3884070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 10 monclient: shutdown 2024-03-29T11:26:00.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.845+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 >> v1:172.21.15.52:6789/0 conn(0x7f3898149dc0 legacy=0x7f389808b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.848+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 shutdown_connections 2024-03-29T11:26:00.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.848+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 >> 172.21.15.52:0/231272293 conn(0x7f38980b35e0 msgr2=0x7f38981409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.848+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 shutdown_connections 2024-03-29T11:26:00.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.848+0000 7f389dc8d640 1 -- 172.21.15.52:0/231272293 wait complete. 2024-03-29T11:26:00.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:00.848+0000 7f389dc8d640 1 librados: shutdown 2024-03-29T11:26:00.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:07.659 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:07.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.827+0000 7f41a614d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:07.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.827+0000 7f41a614d640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.827+0000 7f41a614d640 10 monclient: monmap: 2024-03-29T11:26:07.814 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:07.814 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:07.814 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:07.829033+0000 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:07.829033+0000 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:07.815 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.828+0000 7f41a614d640 1 Processor -- start 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.828+0000 7f41a614d640 1 -- start start 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.828+0000 7f41a614d640 10 monclient: init 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient(hunting): picked mon.noname-c con 0x7f41a00a9930 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient(hunting): picked mon.noname-a con 0x7f41a0149dc0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient(hunting): picked mon.noname-b con 0x7f41a014ff40 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00aae60 con 0x7f41a0149dc0 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00b4310 con 0x7f41a00a9930 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00afdd0 con 0x7f41a014ff40 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.829+0000 7f41a614d640 10 monclient(hunting): authenticate will time out at 1297.970784s 2024-03-29T11:26:07.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.833+0000 7f419f7fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f41a00a9930 0x7f41a0143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:35687/0 (socket says 172.21.15.52:35687) 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.833+0000 7f419f7fe640 1 -- 172.21.15.52:0/2907568916 learned_addr learned my addr 172.21.15.52:0/2907568916 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230656132 0 0) 0x7f41a00b4310 con 0x7f41a00a9930 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4188003520 con 0x7f41a00a9930 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793590836 0 0) 0x7f4188003520 con 0x7f41a00a9930 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.834+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 >> v1:172.21.15.52:6790/0 conn(0x7f41a014ff40 legacy=0x7f41a0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 >> v1:172.21.15.52:6789/0 conn(0x7f41a0149dc0 legacy=0x7f41a014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:26:07.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a014a950 con 0x7f41a00a9930 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4194001690 con 0x7f41a00a9930 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f41a614d640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f41a614d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.835+0000 7f419e7fc640 10 monclient: dump: 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.836+0000 7f419e7fc640 1 -- 172.21.15.52:0/2907568916 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41940019a0 con 0x7f41a00a9930 2024-03-29T11:26:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.836+0000 7f41a614d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.836+0000 7f419e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.836+0000 7f41a614d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.836+0000 7f41a614d640 4 set_mon_vals no callback set 2024-03-29T11:26:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.837+0000 7f41a614d640 10 monclient: shutdown 2024-03-29T11:26:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.837+0000 7f41a614d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.837+0000 7f41a614d640 1 -- 172.21.15.52:0/2907568916 >> v1:172.21.15.79:6789/0 conn(0x7f41a00a9930 legacy=0x7f41a0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.842+0000 7f41a614d640 1 -- 172.21.15.52:0/2907568916 shutdown_connections 2024-03-29T11:26:07.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.842+0000 7f41a614d640 1 -- 172.21.15.52:0/2907568916 >> 172.21.15.52:0/2907568916 conn(0x7f41a00b35e0 msgr2=0x7f41a00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.844+0000 7f41a614d640 1 -- 172.21.15.52:0/2907568916 shutdown_connections 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.845+0000 7f41a614d640 1 -- 172.21.15.52:0/2907568916 wait complete. 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.846+0000 7f41a614d640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.846+0000 7f41a614d640 10 monclient: monmap: 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:07.833 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 1 librados: starting msgr at 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 1 librados: starting objecter 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 1 Processor -- start 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 1 -- start start 2024-03-29T11:26:07.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 1 librados: setting wanted keys 2024-03-29T11:26:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 1 librados: calling monclient init 2024-03-29T11:26:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.847+0000 7f41a614d640 10 monclient: init 2024-03-29T11:26:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.848+0000 7f41a614d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.848+0000 7f41a614d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.848+0000 7f41a614d640 10 monclient(hunting): picked mon.noname-c con 0x7f41a00a9930 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 10 monclient(hunting): picked mon.noname-a con 0x7f41a0149dc0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 10 monclient(hunting): picked mon.noname-b con 0x7f41a014ff40 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a01db840 con 0x7f41a0149dc0 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a006ff20 con 0x7f41a014ff40 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a01db480 con 0x7f41a00a9930 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.849+0000 7f41a614d640 10 monclient(hunting): authenticate will time out at 1297.990818s 2024-03-29T11:26:07.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.853+0000 7f419effd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f41a0149dc0 0x7f41a006f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:48173/0 (socket says 172.21.15.52:48173) 2024-03-29T11:26:07.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.853+0000 7f419effd640 1 -- 172.21.15.52:0/2147417069 learned_addr learned my addr 172.21.15.52:0/2147417069 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:07.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.853+0000 7f4187fff640 10 client.?.objecter ms_handle_connect 0x7f41a014ff40 2024-03-29T11:26:07.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.853+0000 7f4187fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752896409 0 0) 0x7f41a006ff20 con 0x7f41a014ff40 2024-03-29T11:26:07.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:26:07.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f418c004eb0 con 0x7f41a014ff40 2024-03-29T11:26:07.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 client.?.objecter ms_handle_connect 0x7f41a00a9930 2024-03-29T11:26:07.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211821105 0 0) 0x7f41a01db480 con 0x7f41a00a9930 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a006ff20 con 0x7f41a00a9930 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202288410 0 0) 0x7f418c004eb0 con 0x7f41a014ff40 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41a01db480 con 0x7f41a014ff40 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4198001740 con 0x7f41a014ff40 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.854+0000 7f4187fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient(hunting): dump: 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:07.851 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871747305 0 0) 0x7f41a006ff20 con 0x7f41a00a9930 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f418c004eb0 con 0x7f41a00a9930 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4194001020 con 0x7f41a00a9930 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:07.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient(hunting): dump: 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3078359066 0 0) 0x7f418c004eb0 con 0x7f41a00a9930 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 >> v1:172.21.15.79:6789/0 conn(0x7f41a014ff40 legacy=0x7f41a01e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 >> v1:172.21.15.52:6789/0 conn(0x7f41a0149dc0 legacy=0x7f41a006f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 monclient: found mon.c 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.855+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0072500 con 0x7f41a00a9930 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f4187fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f4187fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f4187fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 10 monclient: _renew_subs 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a006eb00 con 0x7f41a00a9930 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 10 monclient: _renew_subs 2024-03-29T11:26:07.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a006eea0 con 0x7f41a00a9930 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f41a614d640 1 librados: init done 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4194001270 con 0x7f41a00a9930 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f4187fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.856+0000 7f419d7fa640 4 set_mon_vals no callback set 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4194004e10 con 0x7f41a00a9930 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f4187fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f4187fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f4187fff640 10 monclient: dump: 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:07.855 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f41a614d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f41a614d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f41a614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.857+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4180005080 con 0x7f41a00a9930 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.858+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f419401ade0 con 0x7f41a00a9930 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f419effd640 10 monclient: get_auth_request con 0x7f418c06de60 auth_method 0 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f4194084c10 con 0x7f41a00a9930 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f4187fff640 10 client.4475.objecter ms_dispatch 0x7f41a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f4187fff640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f4187fff640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f4187fff640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:26:07.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.859+0000 7f4187fff640 10 client.4475.objecter ms_handle_connect 0x7f418c06de60 2024-03-29T11:26:07.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.867+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4194050320 con 0x7f41a00a9930 2024-03-29T11:26:07.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.867+0000 7f4187fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:07.867+0000 7f4187fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:08.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.200+0000 7f41a614d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.200+0000 7f41a614d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.200+0000 7f41a614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:08.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.200+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4180005330 con 0x7f41a00a9930 2024-03-29T11:26:08.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.201+0000 7f4187fff640 1 -- 172.21.15.52:0/2147417069 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f4194052420 con 0x7f41a00a9930 2024-03-29T11:26:08.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.201+0000 7f4187fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.201+0000 7f4187fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:08.192 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:08.192 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:08.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.208+0000 7f41a614d640 10 librados: watch_flush enter 2024-03-29T11:26:08.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.208+0000 7f41a614d640 10 librados: watch_flush exit 2024-03-29T11:26:08.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.208+0000 7f41a614d640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:26:08.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.208+0000 7f41a614d640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:26:08.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.208+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 >> v1:172.21.15.79:6816/407480584 conn(0x7f418c06de60 legacy=0x7f418c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 10 monclient: shutdown 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 >> v1:172.21.15.52:6790/0 conn(0x7f41a00a9930 legacy=0x7f41a01dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 shutdown_connections 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 >> 172.21.15.52:0/2147417069 conn(0x7f41a00b35e0 msgr2=0x7f41a00b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 shutdown_connections 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 1 -- 172.21.15.52:0/2147417069 wait complete. 2024-03-29T11:26:08.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:08.209+0000 7f41a614d640 1 librados: shutdown 2024-03-29T11:26:08.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:15.027 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:15.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.197+0000 7f7ee9979640 10 monclient: get_monmap_and_config 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.197+0000 7f7ee9979640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.197+0000 7f7ee9979640 10 monclient: monmap: 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:15.198803+0000 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:15.198803+0000 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:15.184 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.198+0000 7f7ee9979640 1 Processor -- start 2024-03-29T11:26:15.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.199+0000 7f7ee9979640 1 -- start start 2024-03-29T11:26:15.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.199+0000 7f7ee9979640 10 monclient: init 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee9979640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee9979640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee9979640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee41457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee9979640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee413f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee2ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f7ee41457f0 0x7f7ee413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:35322/0 (socket says 172.21.15.52:35322) 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee2ffd640 1 -- 172.21.15.52:0/716683810 learned_addr learned my addr 172.21.15.52:0/716683810 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee9979640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee40a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.200+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee40aaef0 con 0x7f7ee413f650 2024-03-29T11:26:15.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee4145bd0 con 0x7f7ee41457f0 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee4145eb0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee9979640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee9979640 10 monclient(hunting): authenticate will time out at 1305.341480s 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028208586 0 0) 0x7f7ee4145bd0 con 0x7f7ee41457f0 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee1ffb640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.201+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ecc003520 con 0x7f7ee41457f0 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289283322 0 0) 0x7f7ee4145eb0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee4145bd0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218705684 0 0) 0x7f7ee40aaef0 con 0x7f7ee413f650 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee4145eb0 con 0x7f7ee413f650 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609068836 0 0) 0x7f7ee4145bd0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.202+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 >> v1:172.21.15.79:6789/0 conn(0x7f7ee41457f0 legacy=0x7f7ee413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 >> v1:172.21.15.52:6789/0 conn(0x7f7ee413f650 legacy=0x7f7ee40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4146340 con 0x7f7ee40a9d70 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7edc0019f0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: dump: 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.195 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.196 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.196 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.196 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.196 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:15.196 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:15.196 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7edc001c10 con 0x7f7ee40a9d70 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 1 -- 172.21.15.52:0/716683810 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7edc001fa0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.203+0000 7f7ee1ffb640 10 monclient: dump: 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.197 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.204+0000 7f7ee9979640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.204+0000 7f7ee9979640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.204+0000 7f7ee9979640 4 set_mon_vals no callback set 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.204+0000 7f7ee9979640 10 monclient: shutdown 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.204+0000 7f7ee9979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.204+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 >> v1:172.21.15.52:6790/0 conn(0x7f7ee40a9d70 legacy=0x7f7ee414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.205+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 shutdown_connections 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.205+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 >> 172.21.15.52:0/716683810 conn(0x7f7ee4066a00 msgr2=0x7f7ee4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.205+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 shutdown_connections 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.205+0000 7f7ee9979640 1 -- 172.21.15.52:0/716683810 wait complete. 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.206+0000 7f7ee9979640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.206+0000 7f7ee9979640 10 monclient: monmap: 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:15.199 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.200 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.200 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:15.200 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:15.200 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:15.200 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.207+0000 7f7ee9979640 1 librados: starting msgr at 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.207+0000 7f7ee9979640 1 librados: starting objecter 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.207+0000 7f7ee9979640 1 Processor -- start 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.208+0000 7f7ee9979640 1 -- start start 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.208+0000 7f7ee9979640 1 librados: setting wanted keys 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.208+0000 7f7ee9979640 1 librados: calling monclient init 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.208+0000 7f7ee9979640 10 monclient: init 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.209+0000 7f7ee9979640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.209+0000 7f7ee9979640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.209+0000 7f7ee9979640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee40a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.209+0000 7f7ee9979640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee413f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:15.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee2ffd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7ee40a9d70 0x7f7ee41408a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:47478/0 (socket says 172.21.15.52:47478) 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee27fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f7ee413f650 0x7f7ee41368b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:36746/0 (socket says 172.21.15.52:36746) 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee2ffd640 1 -- 172.21.15.52:0/835344013 learned_addr learned my addr 172.21.15.52:0/835344013 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee9979640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee41457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee9979640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee41cc020 con 0x7f7ee40a9d70 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee9979640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee4140fb0 con 0x7f7ee413f650 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 client.?.objecter ms_handle_connect 0x7f7ee413f650 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 client.?.objecter ms_handle_connect 0x7f7ee40a9d70 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee9979640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee41376d0 con 0x7f7ee41457f0 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee9979640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ee9979640 10 monclient(hunting): authenticate will time out at 1305.350727s 2024-03-29T11:26:15.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249034573 0 0) 0x7f7ee41cc020 con 0x7f7ee40a9d70 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed0004eb0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454745312 0 0) 0x7f7ee4140fb0 con 0x7f7ee413f650 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed00051d0 con 0x7f7ee413f650 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 client.?.objecter ms_handle_connect 0x7f7ee41457f0 2024-03-29T11:26:15.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.210+0000 7f7ecbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597531486 0 0) 0x7f7ee41376d0 con 0x7f7ee41457f0 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed0005510 con 0x7f7ee41457f0 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4205499032 0 0) 0x7f7ed0004eb0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ee41376d0 con 0x7f7ee40a9d70 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7ed4001830 con 0x7f7ee40a9d70 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): dump: 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:15.207 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264196477 0 0) 0x7f7ed00051d0 con 0x7f7ee413f650 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ed0004eb0 con 0x7f7ee413f650 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7ed80018f0 con 0x7f7ee413f650 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.211+0000 7f7ecbfff640 10 monclient(hunting): dump: 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:15.209 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 755878814 0 0) 0x7f7ed0005510 con 0x7f7ee41457f0 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ed0006380 con 0x7f7ee41457f0 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7edc001930 con 0x7f7ee41457f0 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:15.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient(hunting): dump: 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3697935385 0 0) 0x7f7ed0004eb0 con 0x7f7ee413f650 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 >> v1:172.21.15.52:6790/0 conn(0x7f7ee41457f0 legacy=0x7f7ee4136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 >> v1:172.21.15.52:6789/0 conn(0x7f7ee40a9d70 legacy=0x7f7ee41408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 monclient: found mon.b 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee41403a0 con 0x7f7ee413f650 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ed80035f0 con 0x7f7ee413f650 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7ed80018f0 con 0x7f7ee413f650 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:15.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ecbfff640 10 monclient: dump: 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 10 monclient: _renew_subs 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:15.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee4137a10 con 0x7f7ee413f650 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 10 monclient: _renew_subs 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee413ff80 con 0x7f7ee413f650 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee9979640 1 librados: init done 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.212+0000 7f7ee17fa640 4 set_mon_vals no callback set 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.214+0000 7f7ec9ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.214+0000 7f7ec9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.214+0000 7f7ec9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.214+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4075f10 con 0x7f7ee413f650 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f7ed80038e0 con 0x7f7ee413f650 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f7ed80542b0 con 0x7f7ee413f650 2024-03-29T11:26:15.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ecbfff640 10 client.4426.objecter ms_dispatch 0x7f7ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ecbfff640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ecbfff640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ecbfff640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.215+0000 7f7ee2ffd640 10 monclient: get_auth_request con 0x7f7ed006eae0 auth_method 0 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.216+0000 7f7ecbfff640 10 client.4426.objecter ms_handle_connect 0x7f7ed006eae0 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.222+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ed8050550 con 0x7f7ee413f650 2024-03-29T11:26:15.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.222+0000 7f7ecbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.222+0000 7f7ecbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:15.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.649+0000 7f7ec9ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.649+0000 7f7ec9ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.649+0000 7f7ec9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:15.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.649+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ee4140530 con 0x7f7ee413f650 2024-03-29T11:26:15.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.650+0000 7f7ecbfff640 1 -- 172.21.15.52:0/835344013 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f7ed8050550 con 0x7f7ee413f650 2024-03-29T11:26:15.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.650+0000 7f7ecbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.650+0000 7f7ecbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:15.636 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:15.636 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:15.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.654+0000 7f7ec9ffb640 10 librados: watch_flush enter 2024-03-29T11:26:15.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.654+0000 7f7ec9ffb640 10 librados: watch_flush exit 2024-03-29T11:26:15.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.654+0000 7f7ec9ffb640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:26:15.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.654+0000 7f7ec9ffb640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:26:15.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 >> v1:172.21.15.79:6816/407480584 conn(0x7f7ed006eae0 legacy=0x7f7ed0070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 10 monclient: shutdown 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 >> v1:172.21.15.79:6789/0 conn(0x7f7ee413f650 legacy=0x7f7ee41368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 shutdown_connections 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 >> 172.21.15.52:0/835344013 conn(0x7f7ee4066a00 msgr2=0x7f7ee413a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 shutdown_connections 2024-03-29T11:26:15.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 -- 172.21.15.52:0/835344013 wait complete. 2024-03-29T11:26:15.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:15.655+0000 7f7ec9ffb640 1 librados: shutdown 2024-03-29T11:26:15.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:22.467 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:22.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.646+0000 7fb35dee4640 10 monclient: get_monmap_and_config 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.647+0000 7fb35dee4640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.647+0000 7fb35dee4640 10 monclient: monmap: 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:22.647951+0000 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:22.647951+0000 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:22.633 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.648+0000 7fb35dee4640 1 Processor -- start 2024-03-29T11:26:22.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.648+0000 7fb35dee4640 1 -- start start 2024-03-29T11:26:22.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.648+0000 7fb35dee4640 10 monclient: init 2024-03-29T11:26:22.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:22.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 10 monclient(hunting): picked mon.noname-b con 0x7fb3581457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:22.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 10 monclient(hunting): picked mon.noname-a con 0x7fb35813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 10 monclient(hunting): picked mon.noname-c con 0x7fb3580a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb3577fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb3581457f0 0x7fb35813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56312/0 (socket says 172.21.15.52:56312) 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb3577fe640 1 -- 172.21.15.52:0/1279629929 learned_addr learned my addr 172.21.15.52:0/1279629929 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3580aaef0 con 0x7fb35813f650 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.649+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358145bd0 con 0x7fb3580a9d70 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.650+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb358145eb0 con 0x7fb3581457f0 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.650+0000 7fb35dee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.636 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.650+0000 7fb35dee4640 10 monclient(hunting): authenticate will time out at 1312.790160s 2024-03-29T11:26:22.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.650+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137074136 0 0) 0x7fb358145bd0 con 0x7fb3580a9d70 2024-03-29T11:26:22.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb344003520 con 0x7fb3580a9d70 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666512675 0 0) 0x7fb3580aaef0 con 0x7fb35813f650 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb358145bd0 con 0x7fb35813f650 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190091933 0 0) 0x7fb358145eb0 con 0x7fb3581457f0 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3580aaef0 con 0x7fb3581457f0 2024-03-29T11:26:22.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438407872 0 0) 0x7fb344003520 con 0x7fb3580a9d70 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 >> v1:172.21.15.52:6790/0 conn(0x7fb3581457f0 legacy=0x7fb35813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 >> v1:172.21.15.52:6789/0 conn(0x7fb35813f650 legacy=0x7fb3580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 monclient: found mon.noname-c 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358146340 con 0x7fb3580a9d70 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb35dee4640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb350001a10 con 0x7fb3580a9d70 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb35dee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.651+0000 7fb3567fc640 10 monclient: dump: 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb35dee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb350001c30 con 0x7fb3580a9d70 2024-03-29T11:26:22.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb3567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb3567fc640 1 -- 172.21.15.52:0/1279629929 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb350001fc0 con 0x7fb3580a9d70 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb3567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb3567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb3567fc640 10 monclient: dump: 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb35dee4640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:22.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.652+0000 7fb35dee4640 4 set_mon_vals no callback set 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.654+0000 7fb35dee4640 10 monclient: shutdown 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.654+0000 7fb35dee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.654+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 >> v1:172.21.15.79:6789/0 conn(0x7fb3580a9d70 legacy=0x7fb35814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.654+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 shutdown_connections 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.654+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 >> 172.21.15.52:0/1279629929 conn(0x7fb358066a00 msgr2=0x7fb358066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.655+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 shutdown_connections 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.655+0000 7fb35dee4640 1 -- 172.21.15.52:0/1279629929 wait complete. 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.656+0000 7fb35dee4640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.656+0000 7fb35dee4640 10 monclient: monmap: 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:22.651 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.656+0000 7fb35dee4640 1 librados: starting msgr at 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.656+0000 7fb35dee4640 1 librados: starting objecter 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.656+0000 7fb35dee4640 1 Processor -- start 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.657+0000 7fb35dee4640 1 -- start start 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.657+0000 7fb35dee4640 1 librados: setting wanted keys 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.657+0000 7fb35dee4640 1 librados: calling monclient init 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.657+0000 7fb35dee4640 10 monclient: init 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient(hunting): picked mon.noname-a con 0x7fb3580a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient(hunting): picked mon.noname-c con 0x7fb35813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb356ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb35813f650 0x7fb358137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56730/0 (socket says 172.21.15.52:56730) 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb356ffd640 1 -- 172.21.15.52:0/2703296577 learned_addr learned my addr 172.21.15.52:0/2703296577 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:22.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient(hunting): picked mon.noname-b con 0x7fb3581457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35813ae10 con 0x7fb3580a9d70 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb34ffff640 10 client.?.objecter ms_handle_connect 0x7fb35813f650 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb34ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35813a1c0 con 0x7fb3581457f0 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb34ffff640 10 client.?.objecter ms_handle_connect 0x7fb3581457f0 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb34ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35813bc30 con 0x7fb35813f650 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.658+0000 7fb35dee4640 10 monclient(hunting): authenticate will time out at 1312.799052s 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870584367 0 0) 0x7fb35813bc30 con 0x7fb35813f650 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb338004eb0 con 0x7fb35813f650 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752374478 0 0) 0x7fb35813a1c0 con 0x7fb3581457f0 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.659+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35813bc30 con 0x7fb3581457f0 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 client.?.objecter ms_handle_connect 0x7fb3580a9d70 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1193749409 0 0) 0x7fb338004eb0 con 0x7fb35813f650 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb35813a1c0 con 0x7fb35813f650 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb3400019b0 con 0x7fb35813f650 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): dump: 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887890525 0 0) 0x7fb35813bc30 con 0x7fb3581457f0 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb338004eb0 con 0x7fb3581457f0 2024-03-29T11:26:22.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb350001950 con 0x7fb3581457f0 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient(hunting): dump: 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:22.656 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4170274827 0 0) 0x7fb338004eb0 con 0x7fb3581457f0 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 >> v1:172.21.15.52:6790/0 conn(0x7fb35813f650 legacy=0x7fb358137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 >> v1:172.21.15.52:6789/0 conn(0x7fb3580a9d70 legacy=0x7fb358139ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 monclient: found mon.b 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35806df30 con 0x7fb3581457f0 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.660+0000 7fb34ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb350003c20 con 0x7fb3581457f0 2024-03-29T11:26:22.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb34ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb350003f30 con 0x7fb3581457f0 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb34ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb34ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb355ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.661+0000 7fb34ffff640 10 monclient: dump: 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:22.658 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 10 monclient: _renew_subs 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35813a8c0 con 0x7fb3581457f0 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 10 monclient: _renew_subs 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb358138730 con 0x7fb3581457f0 2024-03-29T11:26:22.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.662+0000 7fb35dee4640 1 librados: init done 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.665+0000 7fb35dee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.665+0000 7fb35dee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.665+0000 7fb35dee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.665+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb358075f10 con 0x7fb3581457f0 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.665+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fb350019f00 con 0x7fb3581457f0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.666+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fb350083c20 con 0x7fb3581457f0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.666+0000 7fb34ffff640 10 client.4432.objecter ms_dispatch 0x7fb358000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.666+0000 7fb34ffff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.666+0000 7fb34ffff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.666+0000 7fb34ffff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.666+0000 7fb3577fe640 10 monclient: get_auth_request con 0x7fb33806dec0 auth_method 0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.667+0000 7fb34ffff640 10 client.4432.objecter ms_handle_connect 0x7fb33806dec0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.672+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb35004f500 con 0x7fb3581457f0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.672+0000 7fb34ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:22.672+0000 7fb34ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:23.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.096+0000 7fb35dee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.096+0000 7fb35dee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.096+0000 7fb35dee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:23.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.096+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb358141370 con 0x7fb3581457f0 2024-03-29T11:26:23.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.097+0000 7fb34ffff640 1 -- 172.21.15.52:0/2703296577 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fb350051430 con 0x7fb3581457f0 2024-03-29T11:26:23.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.097+0000 7fb34ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.097+0000 7fb34ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:23.083 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:23.083 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:23.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.100+0000 7fb35dee4640 10 librados: watch_flush enter 2024-03-29T11:26:23.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 10 librados: watch_flush exit 2024-03-29T11:26:23.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T11:26:23.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 10 client.4432.objecter successfully canceled tick 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 >> v1:172.21.15.79:6816/407480584 conn(0x7fb33806dec0 legacy=0x7fb338070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 10 monclient: shutdown 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 >> v1:172.21.15.79:6789/0 conn(0x7fb3581457f0 legacy=0x7fb358141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 shutdown_connections 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.101+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 >> 172.21.15.52:0/2703296577 conn(0x7fb358066a00 msgr2=0x7fb35813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.102+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 shutdown_connections 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.102+0000 7fb35dee4640 1 -- 172.21.15.52:0/2703296577 wait complete. 2024-03-29T11:26:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:23.102+0000 7fb35dee4640 1 librados: shutdown 2024-03-29T11:26:23.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:29.910 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.075+0000 7f613b74d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.075+0000 7f613b74d640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.075+0000 7f613b74d640 10 monclient: monmap: 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:30.076152+0000 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:30.076152+0000 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.076+0000 7f613b74d640 1 Processor -- start 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.076+0000 7f613b74d640 1 -- start start 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.076+0000 7f613b74d640 10 monclient: init 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.077+0000 7f613b74d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.077+0000 7f613b74d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:30.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.077+0000 7f613b74d640 10 monclient(hunting): picked mon.noname-b con 0x7f61341422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 10 monclient(hunting): picked mon.noname-c con 0x7f6134142690 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 10 monclient(hunting): picked mon.noname-a con 0x7f61340b17d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6134142c80 con 0x7f61340b17d0 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61340aae40 con 0x7f6134142690 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613413fbd0 con 0x7f61341422b0 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.078+0000 7f613b74d640 10 monclient(hunting): authenticate will time out at 1320.218343s 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.079+0000 7f6139cea640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f61340b17d0 0x7f61340b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55252/0 (socket says 172.21.15.52:55252) 2024-03-29T11:26:30.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.079+0000 7f6139cea640 1 -- 172.21.15.52:0/2215728054 learned_addr learned my addr 172.21.15.52:0/2215728054 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.081+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867850194 0 0) 0x7f6134142c80 con 0x7f61340b17d0 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.081+0000 7f6133fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.081+0000 7f6133fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.081+0000 7f6133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.081+0000 7f6133fff640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.081+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6118003520 con 0x7f61340b17d0 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340547009 0 0) 0x7f61340aae40 con 0x7f6134142690 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6134142c80 con 0x7f6134142690 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2074346840 0 0) 0x7f6118003520 con 0x7f61340b17d0 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.082+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 >> v1:172.21.15.52:6790/0 conn(0x7f61341422b0 legacy=0x7f613413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:30.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 >> v1:172.21.15.79:6789/0 conn(0x7f6134142690 legacy=0x7f61340b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 1 monclient: found mon.noname-a 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6134151940 con 0x7f61340b17d0 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f612c001a10 con 0x7f61340b17d0 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.083+0000 7f6133fff640 10 monclient: dump: 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:30.073 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f612c001bf0 con 0x7f61340b17d0 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f6133fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f6133fff640 1 -- 172.21.15.52:0/2215728054 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f612c001fc0 con 0x7f61340b17d0 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f6133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f6133fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f6133fff640 10 monclient: dump: 2024-03-29T11:26:30.074 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f613b74d640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f613b74d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:30.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.084+0000 7f613b74d640 4 set_mon_vals no callback set 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.085+0000 7f613b74d640 10 monclient: shutdown 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.085+0000 7f613b74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.085+0000 7f613b74d640 1 -- 172.21.15.52:0/2215728054 >> v1:172.21.15.52:6789/0 conn(0x7f61340b17d0 legacy=0x7f61340b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.085+0000 7f613b74d640 1 -- 172.21.15.52:0/2215728054 shutdown_connections 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.085+0000 7f613b74d640 1 -- 172.21.15.52:0/2215728054 >> 172.21.15.52:0/2215728054 conn(0x7f6134066a00 msgr2=0x7f6134066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.088+0000 7f613b74d640 1 -- 172.21.15.52:0/2215728054 shutdown_connections 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.088+0000 7f613b74d640 1 -- 172.21.15.52:0/2215728054 wait complete. 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.089+0000 7f613b74d640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.089+0000 7f613b74d640 10 monclient: monmap: 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:30.079 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 1 librados: starting msgr at 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 1 librados: starting objecter 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 1 Processor -- start 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 1 -- start start 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 1 librados: setting wanted keys 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 1 librados: calling monclient init 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.090+0000 7f613b74d640 10 monclient: init 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.091+0000 7f613b74d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.091+0000 7f613b74d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.091+0000 7f613b74d640 10 monclient(hunting): picked mon.noname-c con 0x7f61340b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 10 monclient(hunting): picked mon.noname-b con 0x7f61341422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 10 monclient(hunting): picked mon.noname-a con 0x7f6134142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613412db70 con 0x7f6134142690 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61341318e0 con 0x7f61341422b0 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61340ac930 con 0x7f61340b17d0 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.092+0000 7f613b74d640 10 monclient(hunting): authenticate will time out at 1320.232371s 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6139cea640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6134142690 0x7f61341e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:34800/0 (socket says 172.21.15.52:34800) 2024-03-29T11:26:30.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6139cea640 1 -- 172.21.15.52:0/3496174977 learned_addr learned my addr 172.21.15.52:0/3496174977 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 10 client.?.objecter ms_handle_connect 0x7f6134142690 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596904725 0 0) 0x7f6134151940 con 0x7f6134142690 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:26:30.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.093+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f611c004eb0 con 0x7f6134142690 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 979457119 0 0) 0x7f611c004eb0 con 0x7f6134142690 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6134151940 con 0x7f6134142690 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f612c001d60 con 0x7f6134142690 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:30.083 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:30.084 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:30.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:30.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1455245946 0 0) 0x7f6134151940 con 0x7f6134142690 2024-03-29T11:26:30.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.094+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 >> v1:172.21.15.52:6790/0 conn(0x7f61340b17d0 legacy=0x7f61340b6e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:30.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 >> v1:172.21.15.79:6789/0 conn(0x7f61341422b0 legacy=0x7f613412fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 1 monclient: found mon.a 2024-03-29T11:26:30.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61340acc10 con 0x7f6134142690 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 10 client.?.objecter ms_handle_connect 0x7f61341422b0 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f6131ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f613b74d640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f613b74d640 10 monclient: _renew_subs 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f613b74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.097+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6134132150 con 0x7f6134142690 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.098+0000 7f613b74d640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.098+0000 7f613b74d640 10 monclient: _renew_subs 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.098+0000 7f613b74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.098+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61340ab7b0 con 0x7f6134142690 2024-03-29T11:26:30.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.098+0000 7f613b74d640 1 librados: init done 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f612c002040 con 0x7f6134142690 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6131ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f612c001ec0 con 0x7f6134142690 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6131ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6131ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6131ffb640 10 monclient: dump: 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:30.088 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.099+0000 7f6132ffd640 4 set_mon_vals no callback set 2024-03-29T11:26:30.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.100+0000 7f613b74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.101+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f612c01af80 con 0x7f6134142690 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f612c084e10 con 0x7f6134142690 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f6131ffb640 10 client.4461.objecter ms_dispatch 0x7f6134000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f6131ffb640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f6131ffb640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f6131ffb640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f613b74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f613b74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.102+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6110005080 con 0x7f6134142690 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.103+0000 7f61394e9640 10 monclient: get_auth_request con 0x7f611c06dc60 auth_method 0 2024-03-29T11:26:30.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.104+0000 7f6131ffb640 10 client.4461.objecter ms_handle_connect 0x7f611c06dc60 2024-03-29T11:26:30.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.115+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f612c0506f0 con 0x7f6134142690 2024-03-29T11:26:30.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.116+0000 7f6131ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.116+0000 7f6131ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:30.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.455+0000 7f613b74d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.455+0000 7f613b74d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.455+0000 7f613b74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:26:30.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.455+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6110005330 con 0x7f6134142690 2024-03-29T11:26:30.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.457+0000 7f6131ffb640 1 -- 172.21.15.52:0/3496174977 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f612c052620 con 0x7f6134142690 2024-03-29T11:26:30.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.457+0000 7f6131ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.457+0000 7f6131ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:30.443 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:30.443 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:30.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.460+0000 7f613b74d640 10 librados: watch_flush enter 2024-03-29T11:26:30.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.460+0000 7f613b74d640 10 librados: watch_flush exit 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.460+0000 7f613b74d640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.460+0000 7f613b74d640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.461+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 >> v1:172.21.15.79:6816/407480584 conn(0x7f611c06dc60 legacy=0x7f611c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.461+0000 7f613b74d640 10 monclient: shutdown 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.461+0000 7f613b74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.461+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 >> v1:172.21.15.52:6789/0 conn(0x7f6134142690 legacy=0x7f61341e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.461+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 shutdown_connections 2024-03-29T11:26:30.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.461+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 >> 172.21.15.52:0/3496174977 conn(0x7f6134066a00 msgr2=0x7f61340b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.462+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 shutdown_connections 2024-03-29T11:26:30.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.462+0000 7f613b74d640 1 -- 172.21.15.52:0/3496174977 wait complete. 2024-03-29T11:26:30.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:30.462+0000 7f613b74d640 1 librados: shutdown 2024-03-29T11:26:30.466 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:37.274 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.445+0000 7f44bf59e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.445+0000 7f44bf59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.445+0000 7f44bf59e640 10 monclient: monmap: 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:37.446452+0000 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:37.446452+0000 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:37.432 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:37.433 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.446+0000 7f44bf59e640 1 Processor -- start 2024-03-29T11:26:37.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.446+0000 7f44bf59e640 1 -- start start 2024-03-29T11:26:37.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.446+0000 7f44bf59e640 10 monclient: init 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f44b8038c60 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f44b80324b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f44b80d11e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80caf80 con 0x7f44b80324b0 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b8039850 con 0x7f44b8038c60 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80c78d0 con 0x7f44b80d11e0 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.447+0000 7f44bf59e640 10 monclient(hunting): authenticate will time out at 1327.588051s 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.448+0000 7f44be59c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f44b8038c60 0x7f44b80c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:48520/0 (socket says 172.21.15.52:48520) 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.448+0000 7f44be59c640 1 -- 172.21.15.52:0/3596272478 learned_addr learned my addr 172.21.15.52:0/3596272478 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.448+0000 7f44bdd9b640 1 --1- 172.21.15.52:0/3596272478 >> v1:172.21.15.52:6789/0 conn(0x7f44b80324b0 0x7f44b80cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:54100/0 (socket says 172.21.15.52:54100) 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302984835 0 0) 0x7f44b80c78d0 con 0x7f44b80d11e0 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a8003520 con 0x7f44b80d11e0 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154304403 0 0) 0x7f44b80caf80 con 0x7f44b80324b0 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b80c78d0 con 0x7f44b80324b0 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233542990 0 0) 0x7f44b80c78d0 con 0x7f44b80324b0 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 >> v1:172.21.15.52:6790/0 conn(0x7f44b80d11e0 legacy=0x7f44b80d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 >> v1:172.21.15.79:6789/0 conn(0x7f44b8038c60 legacy=0x7f44b80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:37.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 monclient: found mon.noname-a 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80c7a60 con 0x7f44b80324b0 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.449+0000 7f44bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bf59e640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f44b0001610 con 0x7f44b80324b0 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bd59a640 10 monclient: dump: 2024-03-29T11:26:37.438 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:37.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44b00018a0 con 0x7f44b80324b0 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bf59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.450+0000 7f44bf59e640 4 set_mon_vals no callback set 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bd59a640 1 -- 172.21.15.52:0/3596272478 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f44b0001c30 con 0x7f44b80324b0 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bd59a640 10 monclient: dump: 2024-03-29T11:26:37.441 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bf59e640 10 monclient: shutdown 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.451+0000 7f44bf59e640 1 -- 172.21.15.52:0/3596272478 >> v1:172.21.15.52:6789/0 conn(0x7f44b80324b0 legacy=0x7f44b80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.452+0000 7f44bf59e640 1 -- 172.21.15.52:0/3596272478 shutdown_connections 2024-03-29T11:26:37.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.452+0000 7f44bf59e640 1 -- 172.21.15.52:0/3596272478 >> 172.21.15.52:0/3596272478 conn(0x7f44b8005c20 msgr2=0x7f44b8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.452+0000 7f44bf59e640 1 -- 172.21.15.52:0/3596272478 shutdown_connections 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.452+0000 7f44bf59e640 1 -- 172.21.15.52:0/3596272478 wait complete. 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.453+0000 7f44bf59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.453+0000 7f44bf59e640 10 monclient: monmap: 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:37.443 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 1 librados: starting msgr at 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 1 librados: starting objecter 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 1 Processor -- start 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 1 -- start start 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 1 librados: setting wanted keys 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 1 librados: calling monclient init 2024-03-29T11:26:37.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.454+0000 7f44bf59e640 10 monclient: init 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f44b80324b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f44b8038c60 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f44b80d11e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80b8c30 con 0x7f44b80324b0 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80c4550 con 0x7f44b8038c60 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bdd9b640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f44b8038c60 0x7f44b8182360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:48878/0 (socket says 172.21.15.52:48878) 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bdd9b640 1 -- 172.21.15.52:0/2375633850 learned_addr learned my addr 172.21.15.52:0/2375633850 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:37.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b8036ba0 con 0x7f44b80d11e0 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.455+0000 7f44bf59e640 10 monclient(hunting): authenticate will time out at 1327.595773s 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 client.?.objecter ms_handle_connect 0x7f44b80d11e0 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 client.?.objecter ms_handle_connect 0x7f44b80324b0 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 client.?.objecter ms_handle_connect 0x7f44b8038c60 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688276661 0 0) 0x7f44b8036ba0 con 0x7f44b80d11e0 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a0004eb0 con 0x7f44b80d11e0 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773087934 0 0) 0x7f44b80c4550 con 0x7f44b8038c60 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b8036ba0 con 0x7f44b8038c60 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848519560 0 0) 0x7f44b80b8c30 con 0x7f44b80324b0 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.456+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b80c4550 con 0x7f44b80324b0 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 450225588 0 0) 0x7f44b80c4550 con 0x7f44b80324b0 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44b80b8c30 con 0x7f44b80324b0 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f44b4001510 con 0x7f44b80324b0 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): dump: 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.449 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232834552 0 0) 0x7f44a0004eb0 con 0x7f44b80d11e0 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44b80c4550 con 0x7f44b80d11e0 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f44c004fdd0 con 0x7f44b80d11e0 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 10 monclient(hunting): dump: 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.450 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3926416238 0 0) 0x7f44b8036ba0 con 0x7f44b8038c60 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44a0004eb0 con 0x7f44b8038c60 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2135091631 0 0) 0x7f44b80c4550 con 0x7f44b80d11e0 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 >> v1:172.21.15.79:6789/0 conn(0x7f44b8038c60 legacy=0x7f44b8182360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.457+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 >> v1:172.21.15.52:6789/0 conn(0x7f44b80324b0 legacy=0x7f44b80c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 1 monclient: found mon.c 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b802d170 con 0x7f44b80d11e0 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 10 monclient: _renew_subs 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b8035a40 con 0x7f44b80d11e0 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:37.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 10 monclient: _renew_subs 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b8037450 con 0x7f44b80d11e0 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bf59e640 1 librados: init done 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c004f400 con 0x7f44b80d11e0 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.458+0000 7f44bcd99640 4 set_mon_vals no callback set 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f44c004f730 con 0x7f44b80d11e0 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44aeffd640 10 monclient: dump: 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.453 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44bf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44bf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:37.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.459+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a4005080 con 0x7f44b80d11e0 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f44c0067bf0 con 0x7f44b80d11e0 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f44c00fddb0 con 0x7f44b80d11e0 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44aeffd640 10 client.4493.objecter ms_dispatch 0x7f44c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44aeffd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44aeffd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44aeffd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.461+0000 7f44be59c640 10 monclient: get_auth_request con 0x7f44a006dd50 auth_method 0 2024-03-29T11:26:37.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.462+0000 7f44aeffd640 10 client.4493.objecter ms_handle_connect 0x7f44a006dd50 2024-03-29T11:26:37.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.470+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44c00c96f0 con 0x7f44b80d11e0 2024-03-29T11:26:37.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.470+0000 7f44aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.470+0000 7f44aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:37.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.863+0000 7f44bf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.863+0000 7f44bf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.863+0000 7f44bf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:37.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.863+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44a4005330 con 0x7f44b80d11e0 2024-03-29T11:26:37.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.865+0000 7f44aeffd640 1 -- 172.21.15.52:0/2375633850 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f44c00cb620 con 0x7f44b80d11e0 2024-03-29T11:26:37.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.865+0000 7f44aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.865+0000 7f44aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:37.852 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:37.852 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.867+0000 7f44bf59e640 10 librados: watch_flush enter 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 10 librados: watch_flush exit 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 >> v1:172.21.15.79:6816/407480584 conn(0x7f44a006dd50 legacy=0x7f44a0070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 10 monclient: shutdown 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 >> v1:172.21.15.52:6790/0 conn(0x7f44b80d11e0 legacy=0x7f44b8185a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 shutdown_connections 2024-03-29T11:26:37.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 >> 172.21.15.52:0/2375633850 conn(0x7f44b8005c20 msgr2=0x7f44b80c1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 shutdown_connections 2024-03-29T11:26:37.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.868+0000 7f44bf59e640 1 -- 172.21.15.52:0/2375633850 wait complete. 2024-03-29T11:26:37.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:37.869+0000 7f44bf59e640 1 librados: shutdown 2024-03-29T11:26:37.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:44.676 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:44.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.823+0000 7fb2e9767640 10 monclient: get_monmap_and_config 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.823+0000 7fb2e9767640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.823+0000 7fb2e9767640 10 monclient: monmap: 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:44.824025+0000 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:44.824025+0000 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.824+0000 7fb2e9767640 1 Processor -- start 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.824+0000 7fb2e9767640 1 -- start start 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.824+0000 7fb2e9767640 10 monclient: init 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.824+0000 7fb2e9767640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.824+0000 7fb2e9767640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.824+0000 7fb2e9767640 10 monclient(hunting): picked mon.noname-a con 0x7fb2dc0389e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e27fc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb2dc038dc0 0x7fb2dc0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37862/0 (socket says 172.21.15.52:37862) 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e27fc640 1 -- 172.21.15.52:0/425851677 learned_addr learned my addr 172.21.15.52:0/425851677 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 10 monclient(hunting): picked mon.noname-b con 0x7fb2dc038dc0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 10 monclient(hunting): picked mon.noname-c con 0x7fb2dc030660 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0c54c0 con 0x7fb2dc0389e0 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc036430 con 0x7fb2dc030660 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc039840 con 0x7fb2dc038dc0 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e9767640 10 monclient(hunting): authenticate will time out at 1334.965530s 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048889039 0 0) 0x7fb2dc039840 con 0x7fb2dc038dc0 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e1ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.825+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2cc003520 con 0x7fb2dc038dc0 2024-03-29T11:26:44.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.826+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223947204 0 0) 0x7fb2cc003520 con 0x7fb2dc038dc0 2024-03-29T11:26:44.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.826+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 >> v1:172.21.15.79:6789/0 conn(0x7fb2dc030660 legacy=0x7fb2dc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.826+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 >> v1:172.21.15.52:6789/0 conn(0x7fb2dc0389e0 legacy=0x7fb2dc033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc035690 con 0x7fb2dc038dc0 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb2d8001690 con 0x7fb2dc038dc0 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:44.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: dump: 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e9767640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e9767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2d8001860 con 0x7fb2dc038dc0 2024-03-29T11:26:44.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 1 -- 172.21.15.52:0/425851677 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb2d8001c70 con 0x7fb2dc038dc0 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e1ffb640 10 monclient: dump: 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:44.819 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e9767640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.828+0000 7fb2e9767640 4 set_mon_vals no callback set 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 10 monclient: shutdown 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 >> v1:172.21.15.52:6790/0 conn(0x7fb2dc038dc0 legacy=0x7fb2dc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 shutdown_connections 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 >> 172.21.15.52:0/425851677 conn(0x7fb2dc005c00 msgr2=0x7fb2dc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 shutdown_connections 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.829+0000 7fb2e9767640 1 -- 172.21.15.52:0/425851677 wait complete. 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.830+0000 7fb2e9767640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.831+0000 7fb2e9767640 10 monclient: monmap: 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.831+0000 7fb2e9767640 1 librados: starting msgr at 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.831+0000 7fb2e9767640 1 librados: starting objecter 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.832+0000 7fb2e9767640 1 Processor -- start 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.832+0000 7fb2e9767640 1 -- start start 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.832+0000 7fb2e9767640 1 librados: setting wanted keys 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.832+0000 7fb2e9767640 1 librados: calling monclient init 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.832+0000 7fb2e9767640 10 monclient: init 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient(hunting): picked mon.noname-b con 0x7fb2dc030660 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient(hunting): picked mon.noname-a con 0x7fb2dc0389e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e2ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fb2dc030660 0x7fb2dc02d910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:56650/0 (socket says 172.21.15.52:56650) 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient(hunting): picked mon.noname-c con 0x7fb2dc0c80c0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc02d150 con 0x7fb2dc0389e0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc02ecc0 con 0x7fb2dc030660 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc02d3f0 con 0x7fb2dc0c80c0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e2ffd640 1 -- 172.21.15.52:0/205448856 learned_addr learned my addr 172.21.15.52:0/205448856 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.833+0000 7fb2e9767640 10 monclient(hunting): authenticate will time out at 1334.974003s 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb2dc030660 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579935825 0 0) 0x7fb2dc02ecc0 con 0x7fb2dc030660 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0004eb0 con 0x7fb2dc030660 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1194344164 0 0) 0x7fb2d0004eb0 con 0x7fb2dc030660 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.834+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2dc02ecc0 con 0x7fb2dc030660 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb2dc0c80c0 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb2d4001960 con 0x7fb2dc030660 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713681432 0 0) 0x7fb2dc02d3f0 con 0x7fb2dc0c80c0 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0004eb0 con 0x7fb2dc0c80c0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 992038028 0 0) 0x7fb2dc02ecc0 con 0x7fb2dc030660 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 >> v1:172.21.15.52:6790/0 conn(0x7fb2dc0c80c0 legacy=0x7fb2dc159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.835+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 >> v1:172.21.15.52:6789/0 conn(0x7fb2dc0389e0 legacy=0x7fb2dc15cbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 1 monclient: found mon.b 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0c8510 con 0x7fb2dc030660 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 10 monclient: _renew_subs 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc02f730 con 0x7fb2dc030660 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 10 monclient: _renew_subs 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc15a3d0 con 0x7fb2dc030660 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.836+0000 7fb2e9767640 1 librados: init done 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2d4001bc0 con 0x7fb2dc030660 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb2d4003a40 con 0x7fb2dc030660 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2cb7fe640 10 monclient: dump: 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.837+0000 7fb2e17fa640 4 set_mon_vals no callback set 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.838+0000 7fb2e9767640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.838+0000 7fb2e9767640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.838+0000 7fb2e9767640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:44.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.838+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c4005080 con 0x7fb2dc030660 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.839+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fb2d4003ba0 con 0x7fb2dc030660 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2e27fc640 10 monclient: get_auth_request con 0x7fb2d006d9e0 auth_method 0 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fb2d4084b90 con 0x7fb2dc030660 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2cb7fe640 10 client.4444.objecter ms_dispatch 0x7fb2e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2cb7fe640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2cb7fe640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2cb7fe640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.840+0000 7fb2cb7fe640 10 client.4444.objecter ms_handle_connect 0x7fb2d006d9e0 2024-03-29T11:26:44.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.848+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2d40502a0 con 0x7fb2dc030660 2024-03-29T11:26:44.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.849+0000 7fb2cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:44.849+0000 7fb2cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:45.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.169+0000 7fb2e9767640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.169+0000 7fb2e9767640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.169+0000 7fb2e9767640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:26:45.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.169+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2c4005330 con 0x7fb2dc030660 2024-03-29T11:26:45.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.170+0000 7fb2cb7fe640 1 -- 172.21.15.52:0/205448856 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fb2d40523a0 con 0x7fb2dc030660 2024-03-29T11:26:45.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.170+0000 7fb2cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.170+0000 7fb2cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:45.157 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:45.157 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:45.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 10 librados: watch_flush enter 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 10 librados: watch_flush exit 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 10 client.4444.objecter successfully canceled tick 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 >> v1:172.21.15.79:6816/407480584 conn(0x7fb2d006d9e0 legacy=0x7fb2d006fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 10 monclient: shutdown 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.173+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 >> v1:172.21.15.79:6789/0 conn(0x7fb2dc030660 legacy=0x7fb2dc02d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.174+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 shutdown_connections 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.174+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 >> 172.21.15.52:0/205448856 conn(0x7fb2dc005c00 msgr2=0x7fb2dc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.174+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 shutdown_connections 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.174+0000 7fb2e9767640 1 -- 172.21.15.52:0/205448856 wait complete. 2024-03-29T11:26:45.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:45.174+0000 7fb2e9767640 1 librados: shutdown 2024-03-29T11:26:45.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:51.983 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:52.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.133+0000 7f292a0c1640 10 monclient: get_monmap_and_config 2024-03-29T11:26:52.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.133+0000 7f292a0c1640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.133+0000 7f292a0c1640 10 monclient: monmap: 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:52.134020+0000 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:52.134020+0000 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:52.121 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:52.122 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:52.122 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:52.122 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:52.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.134+0000 7f292a0c1640 1 Processor -- start 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.134+0000 7f292a0c1640 1 -- start start 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.134+0000 7f292a0c1640 10 monclient: init 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.134+0000 7f292a0c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f29240a9ac0 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f292413ee90 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f29240af900 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:52.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240afd10 con 0x7f29240af900 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240af6f0 con 0x7f29240a9ac0 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240aac80 con 0x7f292413ee90 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f292a0c1640 10 monclient(hunting): authenticate will time out at 1342.275273s 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f2922ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f292413ee90 0x7f292413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50838/0 (socket says 172.21.15.52:50838) 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f2922ffd640 1 -- 172.21.15.52:0/762948137 learned_addr learned my addr 172.21.15.52:0/762948137 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522805046 0 0) 0x7f29240aac80 con 0x7f292413ee90 2024-03-29T11:26:52.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:52.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f290c003520 con 0x7f292413ee90 2024-03-29T11:26:52.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2174994811 0 0) 0x7f290c003520 con 0x7f292413ee90 2024-03-29T11:26:52.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 >> v1:172.21.15.79:6789/0 conn(0x7f29240a9ac0 legacy=0x7f29240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:52.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.135+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 >> v1:172.21.15.52:6789/0 conn(0x7f29240af900 legacy=0x7f2924149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:52.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:52.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:26:52.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240aaf60 con 0x7f292413ee90 2024-03-29T11:26:52.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2918001690 con 0x7f292413ee90 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: dump: 2024-03-29T11:26:52.146 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:52.147 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:52.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f292a0c1640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:26:52.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f292a0c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:52.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2918001860 con 0x7f292413ee90 2024-03-29T11:26:52.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 1 -- 172.21.15.52:0/762948137 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2918001cb0 con 0x7f292413ee90 2024-03-29T11:26:52.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:52.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f29227fc640 10 monclient: dump: 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:52.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f292a0c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.138+0000 7f292a0c1640 4 set_mon_vals no callback set 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 10 monclient: shutdown 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 1 -- 172.21.15.52:0/762948137 >> v1:172.21.15.52:6790/0 conn(0x7f292413ee90 legacy=0x7f292413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 1 -- 172.21.15.52:0/762948137 shutdown_connections 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 1 -- 172.21.15.52:0/762948137 >> 172.21.15.52:0/762948137 conn(0x7f29240b35e0 msgr2=0x7f29240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 1 -- 172.21.15.52:0/762948137 shutdown_connections 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.139+0000 7f292a0c1640 1 -- 172.21.15.52:0/762948137 wait complete. 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.140+0000 7f292a0c1640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.140+0000 7f292a0c1640 10 monclient: monmap: 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:52.151 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 1 librados: starting msgr at 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 1 librados: starting objecter 2024-03-29T11:26:52.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 1 Processor -- start 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 1 -- start start 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 1 librados: setting wanted keys 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 1 librados: calling monclient init 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.141+0000 7f292a0c1640 10 monclient: init 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.143+0000 7f292a0c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.143+0000 7f292a0c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f29240a9ac0 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f29240af900 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f292413ee90 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2924141df0 con 0x7f29240a9ac0 2024-03-29T11:26:52.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292413c160 con 0x7f292413ee90 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2924142060 con 0x7f29240af900 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f292a0c1640 10 monclient(hunting): authenticate will time out at 1342.284434s 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f2922ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f29240af900 0x7f292413c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:51390/0 (socket says 172.21.15.52:51390) 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f2922ffd640 1 -- 172.21.15.52:0/783347732 learned_addr learned my addr 172.21.15.52:0/783347732 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 client.?.objecter ms_handle_connect 0x7f29240af900 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688843614 0 0) 0x7f2924142060 con 0x7f29240af900 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910004eb0 con 0x7f29240af900 2024-03-29T11:26:52.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 client.?.objecter ms_handle_connect 0x7f292413ee90 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413134061 0 0) 0x7f2910004eb0 con 0x7f29240af900 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2924142060 con 0x7f29240af900 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2918001c30 con 0x7f29240af900 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.144+0000 7f290bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient(hunting): dump: 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.155 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 437754985 0 0) 0x7f2924142060 con 0x7f29240af900 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 >> v1:172.21.15.79:6789/0 conn(0x7f292413ee90 legacy=0x7f292413cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 >> v1:172.21.15.52:6789/0 conn(0x7f29240a9ac0 legacy=0x7f292412b710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 1 monclient: found mon.c 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:52.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292413f990 con 0x7f29240af900 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.145+0000 7f290bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.146+0000 7f292a0c1640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.146+0000 7f292a0c1640 10 monclient: _renew_subs 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.146+0000 7f292a0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.146+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2924142480 con 0x7f29240af900 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f292a0c1640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f292a0c1640 10 monclient: _renew_subs 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f292a0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2924141030 con 0x7f29240af900 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f292a0c1640 1 librados: init done 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2918004b00 con 0x7f29240af900 2024-03-29T11:26:52.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f290bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2918004ed0 con 0x7f29240af900 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f290bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f290bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f290bfff640 10 monclient: dump: 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.158 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:52.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.147+0000 7f2921ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f292a0c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f292a0c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f292a0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2904005080 con 0x7f29240af900 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f2918005030 con 0x7f29240af900 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f2918084c60 con 0x7f29240af900 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f290bfff640 10 client.4505.objecter ms_dispatch 0x7f2924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f290bfff640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f290bfff640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.149+0000 7f290bfff640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.150+0000 7f29237fe640 10 monclient: get_auth_request con 0x7f291006db80 auth_method 0 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.151+0000 7f290bfff640 10 client.4505.objecter ms_handle_connect 0x7f291006db80 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.155+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2918050540 con 0x7f29240af900 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.155+0000 7f290bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.155+0000 7f290bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:52.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.473+0000 7f292a0c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.473+0000 7f292a0c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.473+0000 7f292a0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:52.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.473+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2904005330 con 0x7f29240af900 2024-03-29T11:26:52.461 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:52.461 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.474+0000 7f290bfff640 1 -- 172.21.15.52:0/783347732 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f2918052470 con 0x7f29240af900 2024-03-29T11:26:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.474+0000 7f290bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.474+0000 7f290bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:52.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 10 librados: watch_flush enter 2024-03-29T11:26:52.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 10 librados: watch_flush exit 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 >> v1:172.21.15.79:6816/407480584 conn(0x7f291006db80 legacy=0x7f2910070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 10 monclient: shutdown 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.477+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 >> v1:172.21.15.52:6790/0 conn(0x7f29240af900 legacy=0x7f292413c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.478+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 shutdown_connections 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.478+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 >> 172.21.15.52:0/783347732 conn(0x7f29240b35e0 msgr2=0x7f29240adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.478+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 shutdown_connections 2024-03-29T11:26:52.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.478+0000 7f292a0c1640 1 -- 172.21.15.52:0/783347732 wait complete. 2024-03-29T11:26:52.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:52.478+0000 7f292a0c1640 1 librados: shutdown 2024-03-29T11:26:52.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:59.288 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:59.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.449+0000 7f02253b5640 10 monclient: get_monmap_and_config 2024-03-29T11:26:59.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.449+0000 7f02253b5640 10 monclient: build_initial_monmap 2024-03-29T11:26:59.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.449+0000 7f02253b5640 10 monclient: monmap: 2024-03-29T11:26:59.439 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:59.439 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:59.439 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:26:59.450868+0000 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:26:59.450868+0000 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.450+0000 7f02253b5640 1 Processor -- start 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 1 -- start start 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 10 monclient: init 2024-03-29T11:26:59.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:59.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:59.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 10 monclient(hunting): picked mon.noname-b con 0x7f02201457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:59.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 10 monclient(hunting): picked mon.noname-a con 0x7f022013f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:59.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 10 monclient(hunting): picked mon.noname-c con 0x7f02200a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:59.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.451+0000 7f02253b5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02200aaef0 con 0x7f022013f650 2024-03-29T11:26:59.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.452+0000 7f02253b5640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0220145bd0 con 0x7f02200a9d70 2024-03-29T11:26:59.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.452+0000 7f02253b5640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0220145eb0 con 0x7f02201457f0 2024-03-29T11:26:59.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.452+0000 7f02253b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:59.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.452+0000 7f02253b5640 10 monclient(hunting): authenticate will time out at 1349.592091s 2024-03-29T11:26:59.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.455+0000 7f021effd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f02201457f0 0x7f022013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56471/0 (socket says 172.21.15.52:56471) 2024-03-29T11:26:59.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.455+0000 7f021effd640 1 -- 172.21.15.52:0/3927491792 learned_addr learned my addr 172.21.15.52:0/3927491792 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665509945 0 0) 0x7f0220145eb0 con 0x7f02201457f0 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0208003520 con 0x7f02201457f0 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113294915 0 0) 0x7f02200aaef0 con 0x7f022013f650 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:59.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.458+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220145eb0 con 0x7f022013f650 2024-03-29T11:26:59.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2629431906 0 0) 0x7f0220145eb0 con 0x7f022013f650 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 >> v1:172.21.15.52:6790/0 conn(0x7f02201457f0 legacy=0x7f022013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 >> v1:172.21.15.79:6789/0 conn(0x7f02200a9d70 legacy=0x7f022014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 1 monclient: found mon.noname-a 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220146340 con 0x7f022013f650 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f02253b5640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f02253b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f02180018a0 con 0x7f022013f650 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.461+0000 7f021dffb640 10 monclient: dump: 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f02253b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0218001a80 con 0x7f022013f650 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f021dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:59.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f021dffb640 1 -- 172.21.15.52:0/3927491792 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f02180018a0 con 0x7f022013f650 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f021dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f021dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f021dffb640 10 monclient: dump: 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f02253b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:59.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.462+0000 7f02253b5640 4 set_mon_vals no callback set 2024-03-29T11:26:59.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.463+0000 7f02253b5640 10 monclient: shutdown 2024-03-29T11:26:59.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.463+0000 7f02253b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.463+0000 7f02253b5640 1 -- 172.21.15.52:0/3927491792 >> v1:172.21.15.52:6789/0 conn(0x7f022013f650 legacy=0x7f02200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.464+0000 7f02253b5640 1 -- 172.21.15.52:0/3927491792 shutdown_connections 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.464+0000 7f02253b5640 1 -- 172.21.15.52:0/3927491792 >> 172.21.15.52:0/3927491792 conn(0x7f0220066a00 msgr2=0x7f0220066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.464+0000 7f02253b5640 1 -- 172.21.15.52:0/3927491792 shutdown_connections 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.465+0000 7f02253b5640 1 -- 172.21.15.52:0/3927491792 wait complete. 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.465+0000 7f02253b5640 10 monclient: build_initial_monmap 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.465+0000 7f02253b5640 10 monclient: monmap: 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:26:59.455 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.466+0000 7f02253b5640 1 librados: starting msgr at 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.466+0000 7f02253b5640 1 librados: starting objecter 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.466+0000 7f02253b5640 1 Processor -- start 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.468+0000 7f02253b5640 1 -- start start 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.468+0000 7f02253b5640 1 librados: setting wanted keys 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.468+0000 7f02253b5640 1 librados: calling monclient init 2024-03-29T11:26:59.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.468+0000 7f02253b5640 10 monclient: init 2024-03-29T11:26:59.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:59.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:59.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient(hunting): picked mon.noname-b con 0x7f02200a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:26:59.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient(hunting): picked mon.noname-a con 0x7f022013f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:26:59.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient(hunting): picked mon.noname-c con 0x7f02201457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:26:59.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02201397b0 con 0x7f022013f650 2024-03-29T11:26:59.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0220139a90 con 0x7f02200a9d70 2024-03-29T11:26:59.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022013e820 con 0x7f02201457f0 2024-03-29T11:26:59.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:59.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.469+0000 7f02253b5640 10 monclient(hunting): authenticate will time out at 1349.609871s 2024-03-29T11:26:59.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.476+0000 7f021f7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f02201457f0 0x7f0220132890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56571/0 (socket says 172.21.15.52:56571) 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.476+0000 7f021f7fe640 1 -- 172.21.15.52:0/1911447576 learned_addr learned my addr 172.21.15.52:0/1911447576 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 10 client.?.objecter ms_handle_connect 0x7f022013f650 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984810323 0 0) 0x7f02201397b0 con 0x7f022013f650 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.477+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c004eb0 con 0x7f022013f650 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.478+0000 7f02077fe640 10 client.?.objecter ms_handle_connect 0x7f02201457f0 2024-03-29T11:26:59.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.478+0000 7f02077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.478+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 69834700 0 0) 0x7f020c004eb0 con 0x7f022013f650 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02201397b0 con 0x7f022013f650 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 10 client.?.objecter ms_handle_connect 0x7f02200a9d70 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215946182 0 0) 0x7f022013e820 con 0x7f02201457f0 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.479+0000 7f02077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c004eb0 con 0x7f02201457f0 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703589789 0 0) 0x7f0220139a90 con 0x7f02200a9d70 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022013e820 con 0x7f02200a9d70 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0218001fb0 con 0x7f022013f650 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.483+0000 7f02077fe640 10 monclient(hunting): dump: 2024-03-29T11:26:59.471 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:59.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3154292193 0 0) 0x7f020c004eb0 con 0x7f02201457f0 2024-03-29T11:26:59.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0220139a90 con 0x7f02201457f0 2024-03-29T11:26:59.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0214001870 con 0x7f02201457f0 2024-03-29T11:26:59.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:59.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 10 monclient(hunting): dump: 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3647228617 0 0) 0x7f022013e820 con 0x7f02200a9d70 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.484+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f020c004eb0 con 0x7f02200a9d70 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 720576075 0 0) 0x7f0220139a90 con 0x7f02201457f0 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 >> v1:172.21.15.79:6789/0 conn(0x7f02200a9d70 legacy=0x7f0220130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 >> v1:172.21.15.52:6789/0 conn(0x7f022013f650 legacy=0x7f0220132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 1 monclient: found mon.c 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022013c9d0 con 0x7f02201457f0 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.485+0000 7f02077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 10 monclient: _renew_subs 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022013e410 con 0x7f02201457f0 2024-03-29T11:26:59.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 10 monclient: _renew_subs 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f022013da40 con 0x7f02201457f0 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.486+0000 7f02253b5640 1 librados: init done 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.487+0000 7f02253b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.487+0000 7f02253b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.487+0000 7f02253b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:59.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.487+0000 7f02253b5640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0220139580 con 0x7f02201457f0 2024-03-29T11:26:59.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.491+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0214001d60 con 0x7f02201457f0 2024-03-29T11:26:59.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.492+0000 7f02077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:59.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.492+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f02140037b0 con 0x7f02201457f0 2024-03-29T11:26:59.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.492+0000 7f02077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.492+0000 7f02077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.492+0000 7f02077fe640 10 monclient: dump: 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.493+0000 7f021cff9640 4 set_mon_vals no callback set 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.493+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f02140039d0 con 0x7f02201457f0 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.494+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f0214084950 con 0x7f02201457f0 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.494+0000 7f02077fe640 10 client.4511.objecter ms_dispatch 0x7f0220000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.494+0000 7f02077fe640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.494+0000 7f02077fe640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.494+0000 7f02077fe640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.496+0000 7f021effd640 10 monclient: get_auth_request con 0x7f020c06e0b0 auth_method 0 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.497+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0214050230 con 0x7f02201457f0 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.497+0000 7f02077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.497+0000 7f02077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.497+0000 7f02077fe640 10 client.4511.objecter ms_handle_connect 0x7f020c06e0b0 2024-03-29T11:26:59.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.828+0000 7f02253b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.828+0000 7f02253b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.828+0000 7f02253b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:26:59.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.828+0000 7f02253b5640 1 -- 172.21.15.52:0/1911447576 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0220139fb0 con 0x7f02201457f0 2024-03-29T11:26:59.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.830+0000 7f02077fe640 1 -- 172.21.15.52:0/1911447576 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f0220139fb0 con 0x7f02201457f0 2024-03-29T11:26:59.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.830+0000 7f02077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.830+0000 7f02077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:59.817 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:26:59.817 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:59.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 10 librados: watch_flush enter 2024-03-29T11:26:59.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 10 librados: watch_flush exit 2024-03-29T11:26:59.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:26:59.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:26:59.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 1 -- 172.21.15.52:0/1911447576 >> v1:172.21.15.79:6816/407480584 conn(0x7f020c06e0b0 legacy=0x7f020c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 10 monclient: shutdown 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.833+0000 7f02057fa640 1 -- 172.21.15.52:0/1911447576 >> v1:172.21.15.52:6790/0 conn(0x7f02201457f0 legacy=0x7f0220132890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.834+0000 7f02057fa640 1 -- 172.21.15.52:0/1911447576 shutdown_connections 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.834+0000 7f02057fa640 1 -- 172.21.15.52:0/1911447576 >> 172.21.15.52:0/1911447576 conn(0x7f0220066a00 msgr2=0x7f02200b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.834+0000 7f02057fa640 1 -- 172.21.15.52:0/1911447576 shutdown_connections 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.834+0000 7f02057fa640 1 -- 172.21.15.52:0/1911447576 wait complete. 2024-03-29T11:26:59.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:26:59.834+0000 7f02057fa640 1 librados: shutdown 2024-03-29T11:26:59.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:06.644 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:06.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.804+0000 7f2146be8640 10 monclient: get_monmap_and_config 2024-03-29T11:27:06.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.804+0000 7f2146be8640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.804+0000 7f2146be8640 10 monclient: monmap: 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:06.805239+0000 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:06.805239+0000 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:06.792 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:06.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.805+0000 7f2146be8640 1 Processor -- start 2024-03-29T11:27:06.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.805+0000 7f2146be8640 1 -- start start 2024-03-29T11:27:06.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.805+0000 7f2146be8640 10 monclient: init 2024-03-29T11:27:06.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:06.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient(hunting): picked mon.noname-b con 0x7f2138006e30 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:06.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient(hunting): picked mon.noname-a con 0x7f2138035e00 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:06.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient(hunting): picked mon.noname-c con 0x7f21380c9670 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:06.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138035970 con 0x7f2138035e00 2024-03-29T11:27:06.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138036280 con 0x7f21380c9670 2024-03-29T11:27:06.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138035640 con 0x7f2138006e30 2024-03-29T11:27:06.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.806+0000 7f2146be8640 10 monclient(hunting): authenticate will time out at 1356.946787s 2024-03-29T11:27:06.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.811+0000 7f21463e7640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f21380c9670 0x7f21380d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:47545/0 (socket says 172.21.15.52:47545) 2024-03-29T11:27:06.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.811+0000 7f21463e7640 1 -- 172.21.15.52:0/3592428322 learned_addr learned my addr 172.21.15.52:0/3592428322 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511411720 0 0) 0x7f2138036280 con 0x7f21380c9670 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003520 con 0x7f21380c9670 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561051115 0 0) 0x7f2138035970 con 0x7f2138035e00 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.813+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138036280 con 0x7f2138035e00 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1609096233 0 0) 0x7f2138036280 con 0x7f2138035e00 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 >> v1:172.21.15.52:6790/0 conn(0x7f2138006e30 legacy=0x7f21380cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 >> v1:172.21.15.79:6789/0 conn(0x7f21380c9670 legacy=0x7f21380d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 monclient: found mon.noname-a 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138034e20 con 0x7f2138035e00 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2146be8640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2146be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2130001670 con 0x7f2138035e00 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: dump: 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:06.807 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2146be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21300018c0 con 0x7f2138035e00 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2144be4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2146be8640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.814+0000 7f2146be8640 4 set_mon_vals no callback set 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2144be4640 1 -- 172.21.15.52:0/3592428322 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2130001c50 con 0x7f2138035e00 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2144be4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2144be4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2144be4640 10 monclient: dump: 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.808 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2146be8640 10 monclient: shutdown 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2146be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2146be8640 1 -- 172.21.15.52:0/3592428322 >> v1:172.21.15.52:6789/0 conn(0x7f2138035e00 legacy=0x7f21380c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2146be8640 1 -- 172.21.15.52:0/3592428322 shutdown_connections 2024-03-29T11:27:06.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.815+0000 7f2146be8640 1 -- 172.21.15.52:0/3592428322 >> 172.21.15.52:0/3592428322 conn(0x7f2138005c00 msgr2=0x7f2138006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.816+0000 7f2146be8640 1 -- 172.21.15.52:0/3592428322 shutdown_connections 2024-03-29T11:27:06.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.816+0000 7f2146be8640 1 -- 172.21.15.52:0/3592428322 wait complete. 2024-03-29T11:27:06.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.817+0000 7f2146be8640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.817+0000 7f2146be8640 10 monclient: monmap: 2024-03-29T11:27:06.811 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:06.812 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:06.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.817+0000 7f2146be8640 1 librados: starting msgr at 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.817+0000 7f2146be8640 1 librados: starting objecter 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.817+0000 7f2146be8640 1 Processor -- start 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.820+0000 7f2146be8640 1 -- start start 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.820+0000 7f2146be8640 1 librados: setting wanted keys 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.820+0000 7f2146be8640 1 librados: calling monclient init 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.820+0000 7f2146be8640 10 monclient: init 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.821+0000 7f2146be8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.821+0000 7f2146be8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.821+0000 7f2146be8640 10 monclient(hunting): picked mon.noname-a con 0x7f2138006e30 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:06.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.822+0000 7f2146be8640 10 monclient(hunting): picked mon.noname-c con 0x7f2138035e00 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:06.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.824+0000 7f21453e5640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f2138035e00 0x7f2138166890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39793/0 (socket says 172.21.15.52:39793) 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.824+0000 7f21453e5640 1 -- 172.21.15.52:0/3397546219 learned_addr learned my addr 172.21.15.52:0/3397546219 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f2146be8640 10 monclient(hunting): picked mon.noname-b con 0x7f21380c9670 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380cf2a0 con 0x7f2138006e30 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213802da90 con 0x7f21380c9670 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380cee80 con 0x7f2138035e00 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 client.?.objecter ms_handle_connect 0x7f2138035e00 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 client.?.objecter ms_handle_connect 0x7f21380c9670 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f2146be8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f2146be8640 10 monclient(hunting): authenticate will time out at 1356.965349s 2024-03-29T11:27:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904394440 0 0) 0x7f213802da90 con 0x7f21380c9670 2024-03-29T11:27:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:27:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124004eb0 con 0x7f21380c9670 2024-03-29T11:27:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317406911 0 0) 0x7f21380cee80 con 0x7f2138035e00 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213802da90 con 0x7f2138035e00 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2424295139 0 0) 0x7f2124004eb0 con 0x7f21380c9670 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21380cee80 con 0x7f21380c9670 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f213c001800 con 0x7f21380c9670 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.825+0000 7f21367fc640 10 monclient(hunting): dump: 2024-03-29T11:27:06.821 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:06.822 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757165798 0 0) 0x7f21380cee80 con 0x7f21380c9670 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 >> v1:172.21.15.52:6790/0 conn(0x7f2138035e00 legacy=0x7f2138166890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 >> v1:172.21.15.52:6789/0 conn(0x7f2138006e30 legacy=0x7f21380ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 1 monclient: found mon.b 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138166fa0 con 0x7f21380c9670 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f21367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 10 monclient: _renew_subs 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21381671f0 con 0x7f21380c9670 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:06.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 10 monclient: _renew_subs 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2138033490 con 0x7f21380c9670 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.826+0000 7f2146be8640 1 librados: init done 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f213c001cd0 con 0x7f21380c9670 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f213c004980 con 0x7f21380c9670 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21367fc640 10 monclient: dump: 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:06.825 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.827+0000 7f21377fe640 4 set_mon_vals no callback set 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.828+0000 7f2146be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.828+0000 7f2146be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.828+0000 7f2146be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.828+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2128005080 con 0x7f21380c9670 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.829+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f213c004ae0 con 0x7f21380c9670 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.830+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f213c0848d0 con 0x7f21380c9670 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.830+0000 7f21367fc640 10 client.4459.objecter ms_dispatch 0x7f2140000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.830+0000 7f21367fc640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.830+0000 7f21367fc640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.830+0000 7f21367fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.830+0000 7f2145be6640 10 monclient: get_auth_request con 0x7f212406de70 auth_method 0 2024-03-29T11:27:06.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.831+0000 7f21367fc640 10 client.4459.objecter ms_handle_connect 0x7f212406de70 2024-03-29T11:27:06.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.836+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f213c0501b0 con 0x7f21380c9670 2024-03-29T11:27:06.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.836+0000 7f21367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:06.836+0000 7f21367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.160+0000 7f2146be8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.160+0000 7f2146be8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.160+0000 7f2146be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.160+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2128005330 con 0x7f21380c9670 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.161+0000 7f21367fc640 1 -- 172.21.15.52:0/3397546219 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f213c0520e0 con 0x7f21380c9670 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.161+0000 7f21367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.161+0000 7f21367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:07.152 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:07.152 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:07.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.164+0000 7f2146be8640 10 librados: watch_flush enter 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 10 librados: watch_flush exit 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 >> v1:172.21.15.79:6816/407480584 conn(0x7f212406de70 legacy=0x7f2124070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 10 monclient: shutdown 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 >> v1:172.21.15.79:6789/0 conn(0x7f21380c9670 legacy=0x7f213816cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 shutdown_connections 2024-03-29T11:27:07.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 >> 172.21.15.52:0/3397546219 conn(0x7f2138005c00 msgr2=0x7f2138037b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:07.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.166+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 shutdown_connections 2024-03-29T11:27:07.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.167+0000 7f2146be8640 1 -- 172.21.15.52:0/3397546219 wait complete. 2024-03-29T11:27:07.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:07.167+0000 7f2146be8640 1 librados: shutdown 2024-03-29T11:27:07.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:13.981 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.143+0000 7f9b7d64e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.143+0000 7f9b7d64e640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.143+0000 7f9b7d64e640 10 monclient: monmap: 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:14.144559+0000 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:14.144559+0000 2024-03-29T11:27:14.133 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.144+0000 7f9b7d64e640 1 Processor -- start 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.144+0000 7f9b7d64e640 1 -- start start 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.144+0000 7f9b7d64e640 10 monclient: init 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.145+0000 7f9b7d64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.145+0000 7f9b7d64e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:14.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.145+0000 7f9b7d64e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780b0500 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:14.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.145+0000 7f9b7d64e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b780b08e0 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:14.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b767fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f9b780b08e0 0x7f9b7813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:56923/0 (socket says 172.21.15.52:56923) 2024-03-29T11:27:14.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b767fc640 1 -- 172.21.15.52:0/2051337014 learned_addr learned my addr 172.21.15.52:0/2051337014 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:14.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b7d64e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b780aeca0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:14.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78140620 con 0x7f9b780b0500 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7813daf0 con 0x7f9b780b08e0 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780ae180 con 0x7f9b780aeca0 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b7d64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.148+0000 7f9b7d64e640 10 monclient(hunting): authenticate will time out at 1364.288966s 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75376836 0 0) 0x7f9b7813daf0 con 0x7f9b780b08e0 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60003520 con 0x7f9b780b08e0 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147160388 0 0) 0x7f9b78140620 con 0x7f9b780b0500 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:27:14.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.149+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7813daf0 con 0x7f9b780b0500 2024-03-29T11:27:14.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.150+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1904385948 0 0) 0x7f9b7813daf0 con 0x7f9b780b0500 2024-03-29T11:27:14.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.150+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 >> v1:172.21.15.52:6790/0 conn(0x7f9b780aeca0 legacy=0x7f9b7814b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:14.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 >> v1:172.21.15.79:6789/0 conn(0x7f9b780b08e0 legacy=0x7f9b7813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 1 monclient: found mon.noname-a 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780accf0 con 0x7f9b780b0500 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9b68001670 con 0x7f9b780b0500 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b75ffb640 10 monclient: dump: 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:14.142 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b7d64e640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.151+0000 7f9b7d64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b680017d0 con 0x7f9b780b0500 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b75ffb640 1 -- 172.21.15.52:0/2051337014 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9b68001c20 con 0x7f9b780b0500 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b75ffb640 10 monclient: dump: 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b7d64e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.152+0000 7f9b7d64e640 4 set_mon_vals no callback set 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 10 monclient: shutdown 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 >> v1:172.21.15.52:6789/0 conn(0x7f9b780b0500 legacy=0x7f9b7813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 shutdown_connections 2024-03-29T11:27:14.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 >> 172.21.15.52:0/2051337014 conn(0x7f9b7806b660 msgr2=0x7f9b7806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 shutdown_connections 2024-03-29T11:27:14.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.153+0000 7f9b7d64e640 1 -- 172.21.15.52:0/2051337014 wait complete. 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.154+0000 7f9b7d64e640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.154+0000 7f9b7d64e640 10 monclient: monmap: 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:14.147 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 1 librados: starting msgr at 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 1 librados: starting objecter 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 1 Processor -- start 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 1 -- start start 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 1 librados: setting wanted keys 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 1 librados: calling monclient init 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.155+0000 7f9b7d64e640 10 monclient: init 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b780aeca0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780b0500 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7813bd00 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:14.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b781241f0 con 0x7f9b780b0500 2024-03-29T11:27:14.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78123360 con 0x7f9b7813bd00 2024-03-29T11:27:14.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.156+0000 7f9b7d64e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7813c200 con 0x7f9b780aeca0 2024-03-29T11:27:14.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.157+0000 7f9b7d64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.157+0000 7f9b7d64e640 10 monclient(hunting): authenticate will time out at 1364.297156s 2024-03-29T11:27:14.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.160+0000 7f9b76ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9b780aeca0 0x7f9b78122850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43177/0 (socket says 172.21.15.52:43177) 2024-03-29T11:27:14.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.160+0000 7f9b76ffd640 1 -- 172.21.15.52:0/1751472365 learned_addr learned my addr 172.21.15.52:0/1751472365 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:14.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.160+0000 7f9b5f7fe640 10 client.?.objecter ms_handle_connect 0x7f9b780aeca0 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.160+0000 7f9b5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.160+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51769683 0 0) 0x7f9b7813c200 con 0x7f9b780aeca0 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.160+0000 7f9b5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64004eb0 con 0x7f9b780aeca0 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2620433096 0 0) 0x7f9b64004eb0 con 0x7f9b780aeca0 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7813c200 con 0x7f9b780aeca0 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9b68001630 con 0x7f9b780aeca0 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2213586522 0 0) 0x7f9b7813c200 con 0x7f9b780aeca0 2024-03-29T11:27:14.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 >> v1:172.21.15.79:6789/0 conn(0x7f9b7813bd00 legacy=0x7f9b781d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.161+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 >> v1:172.21.15.52:6789/0 conn(0x7f9b780b0500 legacy=0x7f9b7813b5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 1 monclient: found mon.c 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7813c390 con 0x7f9b780aeca0 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 10 monclient: _renew_subs 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b78122f60 con 0x7f9b780aeca0 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:14.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 10 monclient: _renew_subs 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.163+0000 7f9b7d64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.164+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781d8260 con 0x7f9b780aeca0 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.164+0000 7f9b7d64e640 1 librados: init done 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b680038a0 con 0x7f9b780aeca0 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9b68003c70 con 0x7f9b780aeca0 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b5f7fe640 10 monclient: dump: 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:14.157 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.165+0000 7f9b757fa640 4 set_mon_vals no callback set 2024-03-29T11:27:14.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.166+0000 7f9b7d64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.167+0000 7f9b7d64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.167+0000 7f9b7d64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:14.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.167+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b58005080 con 0x7f9b780aeca0 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.169+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f9b68003dd0 con 0x7f9b780aeca0 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.169+0000 7f9b767fc640 10 monclient: get_auth_request con 0x7f9b6406dc60 auth_method 0 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.170+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f9b68084cd0 con 0x7f9b780aeca0 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.170+0000 7f9b5f7fe640 10 client.4517.objecter ms_dispatch 0x7f9b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.170+0000 7f9b5f7fe640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.170+0000 7f9b5f7fe640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.170+0000 7f9b5f7fe640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T11:27:14.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.170+0000 7f9b5f7fe640 10 client.4517.objecter ms_handle_connect 0x7f9b6406dc60 2024-03-29T11:27:14.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.176+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b680505b0 con 0x7f9b780aeca0 2024-03-29T11:27:14.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.176+0000 7f9b5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.176+0000 7f9b5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:14.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.512+0000 7f9b7d64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.512+0000 7f9b7d64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.512+0000 7f9b7d64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:14.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.512+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b58005330 con 0x7f9b780aeca0 2024-03-29T11:27:14.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.513+0000 7f9b5f7fe640 1 -- 172.21.15.52:0/1751472365 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f9b680524e0 con 0x7f9b780aeca0 2024-03-29T11:27:14.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.513+0000 7f9b5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.513+0000 7f9b5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:14.501 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:14.501 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:14.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 10 librados: watch_flush enter 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 10 librados: watch_flush exit 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 10 client.4517.objecter successfully canceled tick 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 >> v1:172.21.15.79:6816/407480584 conn(0x7f9b6406dc60 legacy=0x7f9b64070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 10 monclient: shutdown 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.517+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 >> v1:172.21.15.52:6790/0 conn(0x7f9b780aeca0 legacy=0x7f9b78122850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.518+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 shutdown_connections 2024-03-29T11:27:14.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.518+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 >> 172.21.15.52:0/1751472365 conn(0x7f9b7806b660 msgr2=0x7f9b780a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.518+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 shutdown_connections 2024-03-29T11:27:14.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.519+0000 7f9b7d64e640 1 -- 172.21.15.52:0/1751472365 wait complete. 2024-03-29T11:27:14.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:14.519+0000 7f9b7d64e640 1 librados: shutdown 2024-03-29T11:27:14.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:21.333 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:21.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.472+0000 7fe1d5693640 10 monclient: get_monmap_and_config 2024-03-29T11:27:21.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.472+0000 7fe1d5693640 10 monclient: build_initial_monmap 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.472+0000 7fe1d5693640 10 monclient: monmap: 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:21.474722+0000 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:21.474722+0000 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.462 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:21.463 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:21.463 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:21.463 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.473+0000 7fe1d5693640 1 Processor -- start 2024-03-29T11:27:21.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.473+0000 7fe1d5693640 1 -- start start 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.473+0000 7fe1d5693640 10 monclient: init 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d01457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d013f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d00a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00aaef0 con 0x7fe1d01457f0 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0145bd0 con 0x7fe1d00a9d70 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0145eb0 con 0x7fe1d013f650 2024-03-29T11:27:21.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1ce7fc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe1d013f650 0x7fe1d00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39912/0 (socket says 172.21.15.52:39912) 2024-03-29T11:27:21.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1ceffd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe1d01457f0 0x7fe1d013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46698/0 (socket says 172.21.15.52:46698) 2024-03-29T11:27:21.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1ce7fc640 1 -- 172.21.15.52:0/1217160273 learned_addr learned my addr 172.21.15.52:0/1217160273 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:21.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:21.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1d5693640 10 monclient(hunting): authenticate will time out at 1371.615379s 2024-03-29T11:27:21.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244660035 0 0) 0x7fe1d0145eb0 con 0x7fe1d013f650 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b8003520 con 0x7fe1d013f650 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797510164 0 0) 0x7fe1d00aaef0 con 0x7fe1d01457f0 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.474+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d0145eb0 con 0x7fe1d01457f0 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.476+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627979273 0 0) 0x7fe1b8003520 con 0x7fe1d013f650 2024-03-29T11:27:21.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.476+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 >> v1:172.21.15.79:6789/0 conn(0x7fe1d00a9d70 legacy=0x7fe1d014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:21.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 >> v1:172.21.15.52:6789/0 conn(0x7fe1d01457f0 legacy=0x7fe1d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:21.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 1 monclient: found mon.noname-b 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d0146340 con 0x7fe1d013f650 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe1c4001690 con 0x7fe1d013f650 2024-03-29T11:27:21.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1d5693640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1d5693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.477+0000 7fe1cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 10 monclient: dump: 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.481 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1d5693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1c40017f0 con 0x7fe1d013f650 2024-03-29T11:27:21.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 1 -- 172.21.15.52:0/1217160273 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe1c4001c40 con 0x7fe1d013f650 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1cdffb640 10 monclient: dump: 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.483 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1d5693640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.478+0000 7fe1d5693640 4 set_mon_vals no callback set 2024-03-29T11:27:21.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 10 monclient: shutdown 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 1 -- 172.21.15.52:0/1217160273 >> v1:172.21.15.52:6790/0 conn(0x7fe1d013f650 legacy=0x7fe1d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 1 -- 172.21.15.52:0/1217160273 shutdown_connections 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 1 -- 172.21.15.52:0/1217160273 >> 172.21.15.52:0/1217160273 conn(0x7fe1d0066a00 msgr2=0x7fe1d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 1 -- 172.21.15.52:0/1217160273 shutdown_connections 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.479+0000 7fe1d5693640 1 -- 172.21.15.52:0/1217160273 wait complete. 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 10 monclient: build_initial_monmap 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 10 monclient: monmap: 2024-03-29T11:27:21.485 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:21.486 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 1 librados: starting msgr at 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 1 librados: starting objecter 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 1 Processor -- start 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 1 -- start start 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 1 librados: setting wanted keys 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 1 librados: calling monclient init 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.480+0000 7fe1d5693640 10 monclient: init 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:21.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d00a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d013f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d01457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0144620 con 0x7fe1d00a9d70 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0141370 con 0x7fe1d013f650 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0141650 con 0x7fe1d01457f0 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1d5693640 10 monclient(hunting): authenticate will time out at 1371.622555s 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1ceffd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe1d00a9d70 0x7fe1d013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:47032/0 (socket says 172.21.15.52:47032) 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1ceffd640 1 -- 172.21.15.52:0/998865502 learned_addr learned my addr 172.21.15.52:0/998865502 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1b77fe640 10 client.?.objecter ms_handle_connect 0x7fe1d00a9d70 2024-03-29T11:27:21.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1b77fe640 10 client.?.objecter ms_handle_connect 0x7fe1d01457f0 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.481+0000 7fe1b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297262649 0 0) 0x7fe1d0141650 con 0x7fe1d01457f0 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc004eb0 con 0x7fe1d01457f0 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797430153 0 0) 0x7fe1d0144620 con 0x7fe1d00a9d70 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.482+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d0141650 con 0x7fe1d00a9d70 2024-03-29T11:27:21.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 883904093 0 0) 0x7fe1bc004eb0 con 0x7fe1d01457f0 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d0144620 con 0x7fe1d01457f0 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe1c8001b70 con 0x7fe1d01457f0 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): dump: 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.490 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125120529 0 0) 0x7fe1d0141650 con 0x7fe1d00a9d70 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1bc004eb0 con 0x7fe1d00a9d70 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe1c0001890 con 0x7fe1d00a9d70 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 10 monclient(hunting): dump: 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 440078642 0 0) 0x7fe1d0144620 con 0x7fe1d01457f0 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.483+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 >> v1:172.21.15.79:6789/0 conn(0x7fe1d013f650 legacy=0x7fe1d01dedd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.484+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 >> v1:172.21.15.52:6789/0 conn(0x7fe1d00a9d70 legacy=0x7fe1d013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 1 monclient: found mon.c 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d013c650 con 0x7fe1d01457f0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 10 monclient: _renew_subs 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d01404b0 con 0x7fe1d01457f0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 10 monclient: _renew_subs 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:21.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d01452c0 con 0x7fe1d01457f0 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1d5693640 1 librados: init done 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1c8002080 con 0x7fe1d01457f0 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.485+0000 7fe1cd7fa640 4 set_mon_vals no callback set 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.486+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe1c8003bb0 con 0x7fe1d01457f0 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.486+0000 7fe1b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.486+0000 7fe1b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.486+0000 7fe1b77fe640 10 monclient: dump: 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:21.494 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.486+0000 7fe1d5693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.487+0000 7fe1d5693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.487+0000 7fe1d5693640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.487+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b0005080 con 0x7fe1d01457f0 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.487+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fe1c8019b80 con 0x7fe1d01457f0 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.488+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fe1c8083ab0 con 0x7fe1d01457f0 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.488+0000 7fe1b77fe640 10 client.4523.objecter ms_dispatch 0x7fe1d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:21.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.488+0000 7fe1b77fe640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.488+0000 7fe1b77fe640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.488+0000 7fe1ceffd640 10 monclient: get_auth_request con 0x7fe1bc06de00 auth_method 0 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.488+0000 7fe1b77fe640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.489+0000 7fe1b77fe640 10 client.4523.objecter ms_handle_connect 0x7fe1bc06de00 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.494+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1c804f390 con 0x7fe1d01457f0 2024-03-29T11:27:21.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.494+0000 7fe1b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.494+0000 7fe1b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.780+0000 7fe1d5693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.780+0000 7fe1d5693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.780+0000 7fe1d5693640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.780+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1b0005330 con 0x7fe1d01457f0 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.781+0000 7fe1b77fe640 1 -- 172.21.15.52:0/998865502 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fe1c80512c0 con 0x7fe1d01457f0 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.781+0000 7fe1b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.781+0000 7fe1b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:21.771 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:21.771 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:21.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.783+0000 7fe1d5693640 10 librados: watch_flush enter 2024-03-29T11:27:21.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.783+0000 7fe1d5693640 10 librados: watch_flush exit 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.783+0000 7fe1d5693640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.783+0000 7fe1d5693640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 >> v1:172.21.15.79:6816/407480584 conn(0x7fe1bc06de00 legacy=0x7fe1bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 10 monclient: shutdown 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 >> v1:172.21.15.52:6790/0 conn(0x7fe1d01457f0 legacy=0x7fe1d01e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 shutdown_connections 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 >> 172.21.15.52:0/998865502 conn(0x7fe1d0066a00 msgr2=0x7fe1d00b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 shutdown_connections 2024-03-29T11:27:21.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 -- 172.21.15.52:0/998865502 wait complete. 2024-03-29T11:27:21.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:21.784+0000 7fe1d5693640 1 librados: shutdown 2024-03-29T11:27:21.787 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:28.594 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.776+0000 7f072f47b640 10 monclient: get_monmap_and_config 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.776+0000 7f072f47b640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.776+0000 7f072f47b640 10 monclient: monmap: 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:28.778861+0000 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:28.778861+0000 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.766 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:28.767 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:28.767 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:28.767 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.778+0000 7f072f47b640 1 Processor -- start 2024-03-29T11:27:28.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.778+0000 7f072f47b640 1 -- start start 2024-03-29T11:27:28.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.778+0000 7f072f47b640 10 monclient: init 2024-03-29T11:27:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient(hunting): picked mon.noname-b con 0x7f07281457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:28.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient(hunting): picked mon.noname-a con 0x7f072813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient(hunting): picked mon.noname-c con 0x7f07280a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07280aaef0 con 0x7f072813f650 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0728145bd0 con 0x7f07280a9d70 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0728145eb0 con 0x7f07281457f0 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072f47b640 10 monclient(hunting): authenticate will time out at 1378.920465s 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072d217640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f07281457f0 0x7f072813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52424/0 (socket says 172.21.15.52:52424) 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.779+0000 7f072d217640 1 -- 172.21.15.52:0/1575256287 learned_addr learned my addr 172.21.15.52:0/1575256287 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982312268 0 0) 0x7f0728145eb0 con 0x7f07281457f0 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0714003520 con 0x7f07281457f0 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549569579 0 0) 0x7f07280aaef0 con 0x7f072813f650 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0728145eb0 con 0x7f072813f650 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42133113 0 0) 0x7f0728145bd0 con 0x7f07280a9d70 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.780+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07280aaef0 con 0x7f07280a9d70 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1519450537 0 0) 0x7f0728145eb0 con 0x7f072813f650 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 >> v1:172.21.15.52:6790/0 conn(0x7f07281457f0 legacy=0x7f072813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 >> v1:172.21.15.79:6789/0 conn(0x7f07280a9d70 legacy=0x7f072814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 monclient: found mon.noname-a 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728146340 con 0x7f072813f650 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0718001670 con 0x7f072813f650 2024-03-29T11:27:28.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: dump: 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f072f47b640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:27:28.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f072f47b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07180017d0 con 0x7f072813f650 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 1 -- 172.21.15.52:0/1575256287 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0718001be0 con 0x7f072813f650 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f072f47b640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f072f47b640 4 set_mon_vals no callback set 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.781+0000 7f071ffff640 10 monclient: dump: 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.781 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.782+0000 7f072f47b640 10 monclient: shutdown 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.782+0000 7f072f47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.782+0000 7f072f47b640 1 -- 172.21.15.52:0/1575256287 >> v1:172.21.15.52:6789/0 conn(0x7f072813f650 legacy=0x7f07280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.782+0000 7f072f47b640 1 -- 172.21.15.52:0/1575256287 shutdown_connections 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.782+0000 7f072f47b640 1 -- 172.21.15.52:0/1575256287 >> 172.21.15.52:0/1575256287 conn(0x7f0728066a00 msgr2=0x7f0728066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.783+0000 7f072f47b640 1 -- 172.21.15.52:0/1575256287 shutdown_connections 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.783+0000 7f072f47b640 1 -- 172.21.15.52:0/1575256287 wait complete. 2024-03-29T11:27:28.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.784+0000 7f072f47b640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.784+0000 7f072f47b640 10 monclient: monmap: 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.784+0000 7f072f47b640 1 librados: starting msgr at 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.784+0000 7f072f47b640 1 librados: starting objecter 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.785+0000 7f072f47b640 1 Processor -- start 2024-03-29T11:27:28.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.785+0000 7f072f47b640 1 -- start start 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.785+0000 7f072f47b640 1 librados: setting wanted keys 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.785+0000 7f072f47b640 1 librados: calling monclient init 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.785+0000 7f072f47b640 10 monclient: init 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient(hunting): picked mon.noname-c con 0x7f07280a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient(hunting): picked mon.noname-a con 0x7f07281457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient(hunting): picked mon.noname-b con 0x7f07281420a0 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072806f660 con 0x7f07281457f0 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07281db6c0 con 0x7f07281420a0 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07281db350 con 0x7f07280a9d70 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.786+0000 7f072f47b640 10 monclient(hunting): authenticate will time out at 1378.928060s 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f072ca16640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f07281457f0 0x7f07281da530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:57698/0 (socket says 172.21.15.52:57698) 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f072ca16640 1 -- 172.21.15.52:0/1599348752 learned_addr learned my addr 172.21.15.52:0/1599348752 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 client.?.objecter ms_handle_connect 0x7f07281420a0 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 client.?.objecter ms_handle_connect 0x7f07281457f0 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 client.?.objecter ms_handle_connect 0x7f07280a9d70 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622776625 0 0) 0x7f07281db6c0 con 0x7f07281420a0 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.787+0000 7f071dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f07281420a0 2024-03-29T11:27:28.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095091933 0 0) 0x7f07281db350 con 0x7f07280a9d70 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07281db6c0 con 0x7f07280a9d70 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760952747 0 0) 0x7f072806f660 con 0x7f07281457f0 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07281db350 con 0x7f07281457f0 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063373582 0 0) 0x7f070c004eb0 con 0x7f07281420a0 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f072806f660 con 0x7f07281420a0 2024-03-29T11:27:28.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2217595221 0 0) 0x7f07281db6c0 con 0x7f07280a9d70 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f07280a9d70 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0720001bf0 con 0x7f07281420a0 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): dump: 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:28.787 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f07240016f0 con 0x7f07280a9d70 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient(hunting): dump: 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:28.788 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1062792153 0 0) 0x7f070c004eb0 con 0x7f07280a9d70 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 >> v1:172.21.15.79:6789/0 conn(0x7f07281420a0 legacy=0x7f07281dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 >> v1:172.21.15.52:6789/0 conn(0x7f07281457f0 legacy=0x7f07281da530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 monclient: found mon.c 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07281dc540 con 0x7f07280a9d70 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f072f47b640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f072f47b640 10 monclient: _renew_subs 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f072f47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07281db910 con 0x7f07280a9d70 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0724001be0 con 0x7f07280a9d70 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f072f47b640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.789+0000 7f071f7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.790+0000 7f072f47b640 10 monclient: _renew_subs 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.790+0000 7f072f47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.790+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f072806e100 con 0x7f07280a9d70 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.790+0000 7f072f47b640 1 librados: init done 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.791+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f0724004980 con 0x7f07280a9d70 2024-03-29T11:27:28.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.791+0000 7f071dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.791+0000 7f071dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.791+0000 7f071dffb640 10 monclient: dump: 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:28.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.791+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f0724004ba0 con 0x7f07280a9d70 2024-03-29T11:27:28.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f072ca16640 10 monclient: get_auth_request con 0x7f070c06df70 auth_method 0 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f072f47b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f072f47b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f072f47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07281dbaa0 con 0x7f07280a9d70 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f0724054690 con 0x7f07280a9d70 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f071dffb640 10 client.4529.objecter ms_dispatch 0x7f0728000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f071dffb640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.792+0000 7f071dffb640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.793+0000 7f071dffb640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.793+0000 7f071dffb640 10 client.4529.objecter ms_handle_connect 0x7f070c06df70 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.802+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07240501a0 con 0x7f07280a9d70 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.802+0000 7f071dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:28.802+0000 7f071dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:29.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.223+0000 7f072f47b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.223+0000 7f072f47b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.223+0000 7f072f47b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:29.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.223+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0728075f10 con 0x7f07280a9d70 2024-03-29T11:27:29.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.224+0000 7f071dffb640 1 -- 172.21.15.52:0/1599348752 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f07240520d0 con 0x7f07280a9d70 2024-03-29T11:27:29.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.224+0000 7f071dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.224+0000 7f071dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:29.212 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:29.212 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 10 librados: watch_flush enter 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 10 librados: watch_flush exit 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 10 client.4529.objecter successfully canceled tick 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 >> v1:172.21.15.79:6816/407480584 conn(0x7f070c06df70 legacy=0x7f070c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 10 monclient: shutdown 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:29.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.228+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 >> v1:172.21.15.52:6790/0 conn(0x7f07280a9d70 legacy=0x7f07281dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.229+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 shutdown_connections 2024-03-29T11:27:29.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.229+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 >> 172.21.15.52:0/1599348752 conn(0x7f0728066a00 msgr2=0x7f07280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:29.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.229+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 shutdown_connections 2024-03-29T11:27:29.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.229+0000 7f072f47b640 1 -- 172.21.15.52:0/1599348752 wait complete. 2024-03-29T11:27:29.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:29.229+0000 7f072f47b640 1 librados: shutdown 2024-03-29T11:27:29.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:36.039 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.201+0000 7fa788de9640 10 monclient: get_monmap_and_config 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.201+0000 7fa788de9640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.201+0000 7fa788de9640 10 monclient: monmap: 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:36.202800+0000 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:36.202800+0000 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:36.190 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:36.191 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:36.191 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:36.191 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:36.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.202+0000 7fa788de9640 1 Processor -- start 2024-03-29T11:27:36.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.202+0000 7fa788de9640 1 -- start start 2024-03-29T11:27:36.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.202+0000 7fa788de9640 10 monclient: init 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.203+0000 7fa788de9640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.203+0000 7fa788de9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.203+0000 7fa788de9640 10 monclient(hunting): picked mon.noname-b con 0x7fa7841422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.203+0000 7fa788de9640 10 monclient(hunting): picked mon.noname-a con 0x7fa784142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa788de9640 10 monclient(hunting): picked mon.noname-c con 0x7fa7840b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa788de9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa784142c80 con 0x7fa784142690 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa788de9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7840aae40 con 0x7fa7840b17d0 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa788de9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78413fbd0 con 0x7fa7841422b0 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa788de9640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa788de9640 10 monclient(hunting): authenticate will time out at 1386.344126s 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa78259c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa7841422b0 0x7fa78413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42614/0 (socket says 172.21.15.52:42614) 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.204+0000 7fa78259c640 1 -- 172.21.15.52:0/261102910 learned_addr learned my addr 172.21.15.52:0/261102910 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.207+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978360623 0 0) 0x7fa78413fbd0 con 0x7fa7841422b0 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.207+0000 7fa78159a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.207+0000 7fa78159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.207+0000 7fa78159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.207+0000 7fa78159a640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:36.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.207+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c003520 con 0x7fa7841422b0 2024-03-29T11:27:36.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199370977 0 0) 0x7fa76c003520 con 0x7fa7841422b0 2024-03-29T11:27:36.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 >> v1:172.21.15.79:6789/0 conn(0x7fa7840b17d0 legacy=0x7fa7840b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:36.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 >> v1:172.21.15.52:6789/0 conn(0x7fa784142690 legacy=0x7fa7840b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:36.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 monclient: found mon.noname-b 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784151940 con 0x7fa7841422b0 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa774001690 con 0x7fa7841422b0 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: dump: 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.199 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa788de9640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:27:36.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa788de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7740018b0 con 0x7fa7841422b0 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 1 -- 172.21.15.52:0/261102910 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa774001c40 con 0x7fa7841422b0 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa78159a640 10 monclient: dump: 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:36.201 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa788de9640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.210+0000 7fa788de9640 4 set_mon_vals no callback set 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.211+0000 7fa788de9640 10 monclient: shutdown 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.211+0000 7fa788de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.211+0000 7fa788de9640 1 -- 172.21.15.52:0/261102910 >> v1:172.21.15.52:6790/0 conn(0x7fa7841422b0 legacy=0x7fa78413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.212+0000 7fa788de9640 1 -- 172.21.15.52:0/261102910 shutdown_connections 2024-03-29T11:27:36.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.212+0000 7fa788de9640 1 -- 172.21.15.52:0/261102910 >> 172.21.15.52:0/261102910 conn(0x7fa784066a00 msgr2=0x7fa784066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.214+0000 7fa788de9640 1 -- 172.21.15.52:0/261102910 shutdown_connections 2024-03-29T11:27:36.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.214+0000 7fa788de9640 1 -- 172.21.15.52:0/261102910 wait complete. 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.215+0000 7fa788de9640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.215+0000 7fa788de9640 10 monclient: monmap: 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:36.211 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.216+0000 7fa788de9640 1 librados: starting msgr at 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.216+0000 7fa788de9640 1 librados: starting objecter 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.216+0000 7fa788de9640 1 Processor -- start 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.217+0000 7fa788de9640 1 -- start start 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.217+0000 7fa788de9640 1 librados: setting wanted keys 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.217+0000 7fa788de9640 1 librados: calling monclient init 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.217+0000 7fa788de9640 10 monclient: init 2024-03-29T11:27:36.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient(hunting): picked mon.noname-c con 0x7fa7840b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient(hunting): picked mon.noname-a con 0x7fa7841422b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient(hunting): picked mon.noname-b con 0x7fa784142690 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78408bdf0 con 0x7fa7841422b0 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7840ac8c0 con 0x7fa784142690 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7840acba0 con 0x7fa7840b17d0 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.219+0000 7fa788de9640 10 monclient(hunting): authenticate will time out at 1386.359457s 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.221+0000 7fa78259c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa7840b17d0 0x7fa7840ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43098/0 (socket says 172.21.15.52:43098) 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.221+0000 7fa78259c640 1 -- 172.21.15.52:0/3465483653 learned_addr learned my addr 172.21.15.52:0/3465483653 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.221+0000 7fa772ffd640 10 client.?.objecter ms_handle_connect 0x7fa7840b17d0 2024-03-29T11:27:36.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.221+0000 7fa772ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.222+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771616746 0 0) 0x7fa7840acba0 con 0x7fa7840b17d0 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.222+0000 7fa772ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.222+0000 7fa772ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.222+0000 7fa772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.222+0000 7fa772ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.222+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa764004eb0 con 0x7fa7840b17d0 2024-03-29T11:27:36.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 client.?.objecter ms_handle_connect 0x7fa7841422b0 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780675147 0 0) 0x7fa764004eb0 con 0x7fa7840b17d0 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7840acba0 con 0x7fa7840b17d0 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843791501 0 0) 0x7fa78408bdf0 con 0x7fa7841422b0 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa764004eb0 con 0x7fa7841422b0 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa774001550 con 0x7fa7840b17d0 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:36.216 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:36.216 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:36.216 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.217 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.217 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.217 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:36.217 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:36.217 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:36.218 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:36.218 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 client.?.objecter ms_handle_connect 0x7fa784142690 2024-03-29T11:27:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.225+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246722216 0 0) 0x7fa7840ac8c0 con 0x7fa784142690 2024-03-29T11:27:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:27:36.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78408bdf0 con 0x7fa784142690 2024-03-29T11:27:36.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3962429646 0 0) 0x7fa7840acba0 con 0x7fa7840b17d0 2024-03-29T11:27:36.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 >> v1:172.21.15.79:6789/0 conn(0x7fa784142690 legacy=0x7fa78412fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 >> v1:172.21.15.52:6789/0 conn(0x7fa7841422b0 legacy=0x7fa78408b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 monclient: found mon.c 2024-03-29T11:27:36.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7840ace80 con 0x7fa7840b17d0 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa774004bf0 con 0x7fa7840b17d0 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa774005000 con 0x7fa7840b17d0 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa772ffd640 10 monclient: dump: 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:36.222 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa773fff640 4 set_mon_vals no callback set 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa788de9640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa788de9640 10 monclient: _renew_subs 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa788de9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.226+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa784132950 con 0x7fa7840b17d0 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.227+0000 7fa788de9640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:36.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.227+0000 7fa788de9640 10 monclient: _renew_subs 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.227+0000 7fa788de9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.227+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa784130520 con 0x7fa7840b17d0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.227+0000 7fa788de9640 1 librados: init done 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.228+0000 7fa788de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.228+0000 7fa788de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.228+0000 7fa788de9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.228+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa784075f10 con 0x7fa7840b17d0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.229+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fa774005160 con 0x7fa7840b17d0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fa774084cc0 con 0x7fa7840b17d0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa772ffd640 10 client.4535.objecter ms_dispatch 0x7fa784000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa772ffd640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa772ffd640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa781d9b640 10 monclient: get_auth_request con 0x7fa764076620 auth_method 0 2024-03-29T11:27:36.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa772ffd640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:27:36.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.230+0000 7fa772ffd640 10 client.4535.objecter ms_handle_connect 0x7fa764076620 2024-03-29T11:27:36.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.238+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7740505a0 con 0x7fa7840b17d0 2024-03-29T11:27:36.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.238+0000 7fa772ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.238+0000 7fa772ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.624+0000 7fa788de9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.624+0000 7fa788de9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.624+0000 7fa788de9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:36.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.624+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa784132ae0 con 0x7fa7840b17d0 2024-03-29T11:27:36.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.625+0000 7fa772ffd640 1 -- 172.21.15.52:0/3465483653 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fa7740524d0 con 0x7fa7840b17d0 2024-03-29T11:27:36.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.625+0000 7fa772ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.625+0000 7fa772ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:36.615 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:36.615 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:36.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 10 librados: watch_flush enter 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 10 librados: watch_flush exit 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 >> v1:172.21.15.79:6816/407480584 conn(0x7fa764076620 legacy=0x7fa764078ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 10 monclient: shutdown 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 >> v1:172.21.15.52:6790/0 conn(0x7fa7840b17d0 legacy=0x7fa7840ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 shutdown_connections 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 >> 172.21.15.52:0/3465483653 conn(0x7fa784066a00 msgr2=0x7fa78413c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.630+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 shutdown_connections 2024-03-29T11:27:36.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.631+0000 7fa788de9640 1 -- 172.21.15.52:0/3465483653 wait complete. 2024-03-29T11:27:36.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:36.631+0000 7fa788de9640 1 librados: shutdown 2024-03-29T11:27:36.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:43.441 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.588+0000 7f91572b3640 10 monclient: get_monmap_and_config 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.588+0000 7f91572b3640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.588+0000 7f91572b3640 10 monclient: monmap: 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:43.589602+0000 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:43.589602+0000 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:43.577 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:43.578 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.589+0000 7f91572b3640 1 Processor -- start 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.589+0000 7f91572b3640 1 -- start start 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.589+0000 7f91572b3640 10 monclient: init 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient(hunting): picked mon.noname-b con 0x7f91501422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient(hunting): picked mon.noname-c con 0x7f9150142690 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient(hunting): picked mon.noname-a con 0x7f91500b17d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150142c80 con 0x7f91500b17d0 2024-03-29T11:27:43.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500aae40 con 0x7f9150142690 2024-03-29T11:27:43.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915013fbd0 con 0x7f91501422b0 2024-03-29T11:27:43.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.590+0000 7f91572b3640 10 monclient(hunting): authenticate will time out at 1393.730400s 2024-03-29T11:27:43.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f915484e640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f9150142690 0x7f91500b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:45870/0 (socket says 172.21.15.52:45870) 2024-03-29T11:27:43.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f915484e640 1 -- 172.21.15.52:0/156541769 learned_addr learned my addr 172.21.15.52:0/156541769 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:43.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095139234 0 0) 0x7f91500aae40 con 0x7f9150142690 2024-03-29T11:27:43.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f9147fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f9147fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f9147fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f9147fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:27:43.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.591+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913c003520 con 0x7f9150142690 2024-03-29T11:27:43.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607378794 0 0) 0x7f9150142c80 con 0x7f91500b17d0 2024-03-29T11:27:43.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91500aae40 con 0x7f91500b17d0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4131709425 0 0) 0x7f91500aae40 con 0x7f91500b17d0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 >> v1:172.21.15.52:6790/0 conn(0x7f91501422b0 legacy=0x7f915013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 >> v1:172.21.15.79:6789/0 conn(0x7f9150142690 legacy=0x7f91500b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 monclient: found mon.noname-a 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150151940 con 0x7f91500b17d0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f91480019f0 con 0x7f91500b17d0 2024-03-29T11:27:43.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: dump: 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f91572b3640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f91572b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:43.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9148001bd0 con 0x7f91500b17d0 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f91572b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 1 -- 172.21.15.52:0/156541769 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9148001fa0 con 0x7f91500b17d0 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f91572b3640 4 set_mon_vals no callback set 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.593+0000 7f9147fff640 10 monclient: dump: 2024-03-29T11:27:43.586 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 10 monclient: shutdown 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 1 -- 172.21.15.52:0/156541769 >> v1:172.21.15.52:6789/0 conn(0x7f91500b17d0 legacy=0x7f91500b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 1 -- 172.21.15.52:0/156541769 shutdown_connections 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 1 -- 172.21.15.52:0/156541769 >> 172.21.15.52:0/156541769 conn(0x7f9150066a00 msgr2=0x7f9150066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 1 -- 172.21.15.52:0/156541769 shutdown_connections 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.594+0000 7f91572b3640 1 -- 172.21.15.52:0/156541769 wait complete. 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 10 monclient: monmap: 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:43.588 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 1 librados: starting msgr at 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 1 librados: starting objecter 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 1 Processor -- start 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 1 -- start start 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 1 librados: setting wanted keys 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 1 librados: calling monclient init 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.595+0000 7f91572b3640 10 monclient: init 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient(hunting): picked mon.noname-a con 0x7f91500b17d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:43.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient(hunting): picked mon.noname-b con 0x7f91501422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient(hunting): picked mon.noname-c con 0x7f9150142690 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91501321a0 con 0x7f91500b17d0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500ac120 con 0x7f91501422b0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150131da0 con 0x7f9150142690 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f91572b3640 10 monclient(hunting): authenticate will time out at 1393.736571s 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f915484e640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f91501422b0 0x7f91500b0a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:46088/0 (socket says 172.21.15.52:46088) 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.596+0000 7f915484e640 1 -- 172.21.15.52:0/2103151589 learned_addr learned my addr 172.21.15.52:0/2103151589 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 client.?.objecter ms_handle_connect 0x7f91501422b0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878414612 0 0) 0x7f91500ac120 con 0x7f91501422b0 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9134004eb0 con 0x7f91501422b0 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353077552 0 0) 0x7f9134004eb0 con 0x7f91501422b0 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91500ac120 con 0x7f91501422b0 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9140001830 con 0x7f91501422b0 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:43.595 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3466938559 0 0) 0x7f91500ac120 con 0x7f91501422b0 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 >> v1:172.21.15.52:6790/0 conn(0x7f9150142690 legacy=0x7f91501e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.597+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 >> v1:172.21.15.52:6789/0 conn(0x7f91500b17d0 legacy=0x7f91501da900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 1 monclient: found mon.b 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91501db010 con 0x7f91501422b0 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 10 monclient: _renew_subs 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9150132bd0 con 0x7f91501422b0 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91400047e0 con 0x7f91501422b0 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9140004c30 con 0x7f91501422b0 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f9145ffb640 10 monclient: dump: 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:43.597 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91477fe640 4 set_mon_vals no callback set 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 10 monclient: _renew_subs 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9150131b00 con 0x7f91501422b0 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.598+0000 7f91572b3640 1 librados: init done 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.600+0000 7f91572b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.600+0000 7f91572b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.600+0000 7f91572b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.600+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9150132350 con 0x7f91501422b0 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f9140004d90 con 0x7f91501422b0 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f9140084ab0 con 0x7f91501422b0 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f9145ffb640 10 client.4477.objecter ms_dispatch 0x7f9150000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f9145ffb640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f9145ffb640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f915504f640 10 monclient: get_auth_request con 0x7f913406dbc0 auth_method 0 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.601+0000 7f9145ffb640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:27:43.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.602+0000 7f9145ffb640 10 client.4477.objecter ms_handle_connect 0x7f913406dbc0 2024-03-29T11:27:43.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.611+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9150132350 con 0x7f91501422b0 2024-03-29T11:27:43.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.611+0000 7f9145ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.611+0000 7f9145ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:43.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.926+0000 7f91572b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.926+0000 7f91572b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.926+0000 7f91572b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:27:43.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.926+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9150132350 con 0x7f91501422b0 2024-03-29T11:27:43.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.927+0000 7f9145ffb640 1 -- 172.21.15.52:0/2103151589 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f9150132350 con 0x7f91501422b0 2024-03-29T11:27:43.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.927+0000 7f9145ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.915 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.927+0000 7f9145ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:43.916 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:43.916 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:43.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.930+0000 7f91572b3640 10 librados: watch_flush enter 2024-03-29T11:27:43.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.930+0000 7f91572b3640 10 librados: watch_flush exit 2024-03-29T11:27:43.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.930+0000 7f91572b3640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:27:43.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.930+0000 7f91572b3640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 >> v1:172.21.15.79:6816/407480584 conn(0x7f913406dbc0 legacy=0x7f9134070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 10 monclient: shutdown 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 >> v1:172.21.15.79:6789/0 conn(0x7f91501422b0 legacy=0x7f91500b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 shutdown_connections 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 >> 172.21.15.52:0/2103151589 conn(0x7f9150066a00 msgr2=0x7f915013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 shutdown_connections 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 -- 172.21.15.52:0/2103151589 wait complete. 2024-03-29T11:27:43.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:43.931+0000 7f91572b3640 1 librados: shutdown 2024-03-29T11:27:43.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:50.743 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:50.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.888+0000 7f12e4bed640 10 monclient: get_monmap_and_config 2024-03-29T11:27:50.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.888+0000 7f12e4bed640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.888+0000 7f12e4bed640 10 monclient: monmap: 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:50.889728+0000 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:50.889728+0000 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:50.878 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:50.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.889+0000 7f12e4bed640 1 Processor -- start 2024-03-29T11:27:50.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.889+0000 7f12e4bed640 1 -- start start 2024-03-29T11:27:50.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.889+0000 7f12e4bed640 10 monclient: init 2024-03-29T11:27:50.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 10 monclient(hunting): picked mon.noname-c con 0x7f12e01457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 10 monclient(hunting): picked mon.noname-a con 0x7f12e013f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 10 monclient(hunting): picked mon.noname-b con 0x7f12e00a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00aaef0 con 0x7f12e013f650 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.891+0000 7f12e4bed640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0145bd0 con 0x7f12e01457f0 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12e4bed640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0145eb0 con 0x7f12e00a9d70 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12e4bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12e4bed640 10 monclient(hunting): authenticate will time out at 1401.032111s 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12ded9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f12e00a9d70 0x7f12e014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58688/0 (socket says 172.21.15.52:58688) 2024-03-29T11:27:50.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12ded9d640 1 -- 172.21.15.52:0/1563131659 learned_addr learned my addr 172.21.15.52:0/1563131659 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83199380 0 0) 0x7f12e0145eb0 con 0x7f12e00a9d70 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c8003520 con 0x7f12e00a9d70 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330174955 0 0) 0x7f12c8003520 con 0x7f12e00a9d70 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 >> v1:172.21.15.79:6789/0 conn(0x7f12e01457f0 legacy=0x7f12e013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:50.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.892+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 >> v1:172.21.15.52:6789/0 conn(0x7f12e013f650 legacy=0x7f12e00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:50.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 1 monclient: found mon.noname-b 2024-03-29T11:27:50.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:27:50.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0146340 con 0x7f12e00a9d70 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f12d8001a10 con 0x7f12e00a9d70 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.894+0000 7f12dd59a640 10 monclient: dump: 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.887 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12e4bed640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12e4bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12d8001be0 con 0x7f12e00a9d70 2024-03-29T11:27:50.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12dd59a640 1 -- 172.21.15.52:0/1563131659 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f12d8002030 con 0x7f12e00a9d70 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12e4bed640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12e4bed640 4 set_mon_vals no callback set 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.895+0000 7f12dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.896+0000 7f12e4bed640 10 monclient: shutdown 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.896+0000 7f12dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.896+0000 7f12dd59a640 10 monclient: dump: 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:50.889 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.896+0000 7f12e4bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.896+0000 7f12e4bed640 1 -- 172.21.15.52:0/1563131659 >> v1:172.21.15.52:6790/0 conn(0x7f12e00a9d70 legacy=0x7f12e014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.897+0000 7f12e4bed640 1 -- 172.21.15.52:0/1563131659 shutdown_connections 2024-03-29T11:27:50.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.897+0000 7f12e4bed640 1 -- 172.21.15.52:0/1563131659 >> 172.21.15.52:0/1563131659 conn(0x7f12e0066a00 msgr2=0x7f12e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.897+0000 7f12e4bed640 1 -- 172.21.15.52:0/1563131659 shutdown_connections 2024-03-29T11:27:50.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.897+0000 7f12e4bed640 1 -- 172.21.15.52:0/1563131659 wait complete. 2024-03-29T11:27:50.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.898+0000 7f12e4bed640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.898+0000 7f12e4bed640 10 monclient: monmap: 2024-03-29T11:27:50.898 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.898+0000 7f12e4bed640 1 librados: starting msgr at 2024-03-29T11:27:50.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.899+0000 7f12e4bed640 1 librados: starting objecter 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.899+0000 7f12e4bed640 1 Processor -- start 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.899+0000 7f12e4bed640 1 -- start start 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.899+0000 7f12e4bed640 1 librados: setting wanted keys 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.899+0000 7f12e4bed640 1 librados: calling monclient init 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.899+0000 7f12e4bed640 10 monclient: init 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:50.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient(hunting): picked mon.noname-a con 0x7f12e00a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient(hunting): picked mon.noname-c con 0x7f12e013f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient(hunting): picked mon.noname-b con 0x7f12e01457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e01422d0 con 0x7f12e00a9d70 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0142590 con 0x7f12e01457f0 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e01d7620 con 0x7f12e013f650 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12e4bed640 10 monclient(hunting): authenticate will time out at 1401.040352s 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12ddd9b640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f12e013f650 0x7f12e01d6f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:59212/0 (socket says 172.21.15.52:59212) 2024-03-29T11:27:50.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12ddd9b640 1 -- 172.21.15.52:0/3162979755 learned_addr learned my addr 172.21.15.52:0/3162979755 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12ceffd640 10 client.?.objecter ms_handle_connect 0x7f12e013f650 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358112904 0 0) 0x7f12e01d7620 con 0x7f12e013f650 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.900+0000 7f12ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0004eb0 con 0x7f12e013f650 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 client.?.objecter ms_handle_connect 0x7f12e01457f0 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376398361 0 0) 0x7f12e0142590 con 0x7f12e01457f0 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e01d7620 con 0x7f12e01457f0 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1384952514 0 0) 0x7f12c0004eb0 con 0x7f12e013f650 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12e0142590 con 0x7f12e013f650 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.901+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f12d4004080 con 0x7f12e013f650 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient(hunting): dump: 2024-03-29T11:27:50.903 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 834798550 0 0) 0x7f12e01d7620 con 0x7f12e01457f0 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12c0004eb0 con 0x7f12e01457f0 2024-03-29T11:27:50.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2419108276 0 0) 0x7f12e0142590 con 0x7f12e013f650 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 >> v1:172.21.15.79:6789/0 conn(0x7f12e01457f0 legacy=0x7f12e01d42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 >> v1:172.21.15.52:6789/0 conn(0x7f12e00a9d70 legacy=0x7f12e0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 monclient: found mon.c 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0140d50 con 0x7f12e013f650 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:27:50.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 10 monclient: _renew_subs 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e0141eb0 con 0x7f12e013f650 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 10 monclient: _renew_subs 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e0142a20 con 0x7f12e013f650 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.902+0000 7f12e4bed640 1 librados: init done 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.903+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12d40041e0 con 0x7f12e013f650 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.903+0000 7f12ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.903+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f12d4001180 con 0x7f12e013f650 2024-03-29T11:27:50.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.903+0000 7f12dcd99640 4 set_mon_vals no callback set 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.904+0000 7f12ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.904+0000 7f12ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.904+0000 7f12ceffd640 10 monclient: dump: 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:50.907 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.905+0000 7f12ccff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.905+0000 7f12ccff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.905+0000 7f12ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.905+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f12d40012e0 con 0x7f12e013f650 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.905+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c4005080 con 0x7f12e013f650 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.906+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f12d4084980 con 0x7f12e013f650 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.906+0000 7f12ceffd640 10 client.4541.objecter ms_dispatch 0x7f12e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.906+0000 7f12ceffd640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.906+0000 7f12ceffd640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.906+0000 7f12ceffd640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.908+0000 7f12de59c640 10 monclient: get_auth_request con 0x7f12c006de50 auth_method 0 2024-03-29T11:27:50.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.909+0000 7f12ceffd640 10 client.4541.objecter ms_handle_connect 0x7f12c006de50 2024-03-29T11:27:50.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.912+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12d4050260 con 0x7f12e013f650 2024-03-29T11:27:50.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.912+0000 7f12ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:50.912+0000 7f12ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:51.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.234+0000 7f12ccff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.234+0000 7f12ccff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.234+0000 7f12ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:51.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.234+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12c4005330 con 0x7f12e013f650 2024-03-29T11:27:51.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.235+0000 7f12ceffd640 1 -- 172.21.15.52:0/3162979755 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f12d4052190 con 0x7f12e013f650 2024-03-29T11:27:51.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.235+0000 7f12ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.235+0000 7f12ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:51.224 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:51.224 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.238+0000 7f12ccff9640 10 librados: watch_flush enter 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.238+0000 7f12ccff9640 10 librados: watch_flush exit 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 >> v1:172.21.15.79:6816/407480584 conn(0x7f12c006de50 legacy=0x7f12c0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 10 monclient: shutdown 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:51.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 >> v1:172.21.15.52:6790/0 conn(0x7f12e013f650 legacy=0x7f12e01d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 shutdown_connections 2024-03-29T11:27:51.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 >> 172.21.15.52:0/3162979755 conn(0x7f12e0066a00 msgr2=0x7f12e00b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:51.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 shutdown_connections 2024-03-29T11:27:51.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 -- 172.21.15.52:0/3162979755 wait complete. 2024-03-29T11:27:51.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:51.239+0000 7f12ccff9640 1 librados: shutdown 2024-03-29T11:27:51.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:58.050 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:58.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.193+0000 7f35a305c640 10 monclient: get_monmap_and_config 2024-03-29T11:27:58.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.193+0000 7f35a305c640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.193+0000 7f35a305c640 10 monclient: monmap: 2024-03-29T11:27:58.182 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:58.182 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:58.182 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:27:58.194016+0000 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:27:58.194016+0000 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:27:58.183 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.194+0000 7f35a305c640 1 Processor -- start 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.194+0000 7f35a305c640 1 -- start start 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.194+0000 7f35a305c640 10 monclient: init 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient(hunting): picked mon.noname-c con 0x7f359c0b4710 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient(hunting): picked mon.noname-b con 0x7f359c0b3970 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient(hunting): picked mon.noname-a con 0x7f359c13f390 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:58.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0af020 con 0x7f359c13f390 2024-03-29T11:27:58.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0b0910 con 0x7f359c0b4710 2024-03-29T11:27:58.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0b0bf0 con 0x7f359c0b3970 2024-03-29T11:27:58.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f35a305c640 10 monclient(hunting): authenticate will time out at 1408.335564s 2024-03-29T11:27:58.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359bfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f359c0b3970 0x7f359c0af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41854/0 (socket says 172.21.15.52:41854) 2024-03-29T11:27:58.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359bfff640 1 -- 172.21.15.52:0/4201971524 learned_addr learned my addr 172.21.15.52:0/4201971524 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274451852 0 0) 0x7f359c0b0bf0 con 0x7f359c0b3970 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.195+0000 7f359b7fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.196+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3588003520 con 0x7f359c0b3970 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.196+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588441664 0 0) 0x7f3588003520 con 0x7f359c0b3970 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.196+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 >> v1:172.21.15.79:6789/0 conn(0x7f359c0b4710 legacy=0x7f359c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.196+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 >> v1:172.21.15.52:6789/0 conn(0x7f359c13f390 legacy=0x7f359c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0aeb00 con 0x7f359c0b3970 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f35a305c640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f35a305c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3584001690 con 0x7f359c0b3970 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: dump: 2024-03-29T11:27:58.193 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f35a305c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3584001960 con 0x7f359c0b3970 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f359b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f35a305c640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.197+0000 7f35a305c640 4 set_mon_vals no callback set 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f359b7fe640 1 -- 172.21.15.52:0/4201971524 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3584001cf0 con 0x7f359c0b3970 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f359b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f359b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f359b7fe640 10 monclient: dump: 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:58.195 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f35a305c640 10 monclient: shutdown 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f35a305c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f35a305c640 1 -- 172.21.15.52:0/4201971524 >> v1:172.21.15.52:6790/0 conn(0x7f359c0b3970 legacy=0x7f359c0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f35a305c640 1 -- 172.21.15.52:0/4201971524 shutdown_connections 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f35a305c640 1 -- 172.21.15.52:0/4201971524 >> 172.21.15.52:0/4201971524 conn(0x7f359c066a00 msgr2=0x7f359c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.198+0000 7f35a305c640 1 -- 172.21.15.52:0/4201971524 shutdown_connections 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.199+0000 7f35a305c640 1 -- 172.21.15.52:0/4201971524 wait complete. 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.199+0000 7f35a305c640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.199+0000 7f35a305c640 10 monclient: monmap: 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:27:58.196 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 1 librados: starting msgr at 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 1 librados: starting objecter 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 1 Processor -- start 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 1 -- start start 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 1 librados: setting wanted keys 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 1 librados: calling monclient init 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 10 monclient: init 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:58.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.200+0000 7f35a305c640 10 monclient(hunting): picked mon.noname-b con 0x7f359c0b4710 addr v1:172.21.15.79:6789/0 2024-03-29T11:27:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 10 monclient(hunting): picked mon.noname-a con 0x7f359c13f390 addr v1:172.21.15.52:6789/0 2024-03-29T11:27:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 10 monclient(hunting): picked mon.noname-c con 0x7f359c132910 addr v1:172.21.15.52:6790/0 2024-03-29T11:27:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c1382b0 con 0x7f359c13f390 2024-03-29T11:27:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c138590 con 0x7f359c0b4710 2024-03-29T11:27:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0ab3a0 con 0x7f359c132910 2024-03-29T11:27:58.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a305c640 10 monclient(hunting): authenticate will time out at 1408.341120s 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a15f9640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f359c132910 0x7f359c1358d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42004/0 (socket says 172.21.15.52:42004) 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35a15f9640 1 -- 172.21.15.52:0/4251070219 learned_addr learned my addr 172.21.15.52:0/4251070219 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 client.?.objecter ms_handle_connect 0x7f359c132910 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 client.?.objecter ms_handle_connect 0x7f359c0b4710 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642646069 0 0) 0x7f359c0ab3a0 con 0x7f359c132910 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f357c004eb0 con 0x7f359c132910 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1261804953 0 0) 0x7f357c004eb0 con 0x7f359c132910 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f359c0ab3a0 con 0x7f359c132910 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f358c001ad0 con 0x7f359c132910 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): dump: 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:58.200 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:58.201 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:58.201 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.201+0000 7f35997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.202+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1583037710 0 0) 0x7f359c0ab3a0 con 0x7f359c132910 2024-03-29T11:27:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.202+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 >> v1:172.21.15.79:6789/0 conn(0x7f359c0b4710 legacy=0x7f359c0a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.202+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 >> v1:172.21.15.52:6789/0 conn(0x7f359c13f390 legacy=0x7f359c0adb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 1 monclient: found mon.c 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0ae220 con 0x7f359c132910 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 10 monclient: _renew_subs 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:58.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c0ae470 con 0x7f359c132910 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 10 monclient: _renew_subs 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c1361c0 con 0x7f359c132910 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.204+0000 7f35a305c640 1 librados: init done 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f358c002060 con 0x7f359c132910 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f35997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f358c004d80 con 0x7f359c132910 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f35997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f35997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f35997fa640 10 monclient: dump: 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:27:58.204 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.205+0000 7f359affd640 4 set_mon_vals no callback set 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.207+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f358c01ad50 con 0x7f359c132910 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f358c084d90 con 0x7f359c132910 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35997fa640 10 client.4547.objecter ms_dispatch 0x7f359c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35997fa640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35997fa640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35a305c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35a305c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35a305c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35997fa640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3580005080 con 0x7f359c132910 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35a0df8640 10 monclient: get_auth_request con 0x7f357c06dbd0 auth_method 0 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.208+0000 7f35997fa640 10 client.4547.objecter ms_handle_connect 0x7f357c06dbd0 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.215+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f358c0504a0 con 0x7f359c132910 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.215+0000 7f35997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.215+0000 7f35997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:58.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35a305c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35a305c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35a305c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:27:58.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3580005330 con 0x7f359c132910 2024-03-29T11:27:58.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35997fa640 1 -- 172.21.15.52:0/4251070219 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f358c0525a0 con 0x7f359c132910 2024-03-29T11:27:58.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.527+0000 7f35997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:58.517 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:27:58.517 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.530+0000 7f35a305c640 10 librados: watch_flush enter 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.530+0000 7f35a305c640 10 librados: watch_flush exit 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.530+0000 7f35a305c640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 >> v1:172.21.15.79:6816/407480584 conn(0x7f357c06dbd0 legacy=0x7f357c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 10 monclient: shutdown 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 >> v1:172.21.15.52:6790/0 conn(0x7f359c132910 legacy=0x7f359c1358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 shutdown_connections 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 >> 172.21.15.52:0/4251070219 conn(0x7f359c066a00 msgr2=0x7f359c0b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 shutdown_connections 2024-03-29T11:27:58.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 -- 172.21.15.52:0/4251070219 wait complete. 2024-03-29T11:27:58.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:27:58.531+0000 7f35a305c640 1 librados: shutdown 2024-03-29T11:27:58.534 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:05.343 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:05.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.500+0000 7f431bd32640 10 monclient: get_monmap_and_config 2024-03-29T11:28:05.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.500+0000 7f431bd32640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.500+0000 7f431bd32640 10 monclient: monmap: 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:05.501432+0000 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:05.501432+0000 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.501+0000 7f431bd32640 1 Processor -- start 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.501+0000 7f431bd32640 1 -- start start 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.501+0000 7f431bd32640 10 monclient: init 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient(hunting): picked mon.noname-a con 0x7f43140a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient(hunting): picked mon.noname-b con 0x7f4314149890 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient(hunting): picked mon.noname-c con 0x7f43140afa30 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140b00c0 con 0x7f43140a9750 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43141464a0 con 0x7f43140afa30 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140ab280 con 0x7f4314149890 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f431bd32640 10 monclient(hunting): authenticate will time out at 1415.643234s 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f43192cd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f4314149890 0x7f4314149da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:48664/0 (socket says 172.21.15.52:48664) 2024-03-29T11:28:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.503+0000 7f43192cd640 1 -- 172.21.15.52:0/3784367461 learned_addr learned my addr 172.21.15.52:0/3784367461 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:05.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.506+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656494397 0 0) 0x7f43140ab280 con 0x7f4314149890 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.506+0000 7f4318acc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.506+0000 7f4318acc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.506+0000 7f4318acc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.506+0000 7f4318acc640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.506+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4300003520 con 0x7f4314149890 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.507+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 268254078 0 0) 0x7f4300003520 con 0x7f4314149890 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.507+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 >> v1:172.21.15.79:6789/0 conn(0x7f43140afa30 legacy=0x7f4314151c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:05.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 >> v1:172.21.15.52:6789/0 conn(0x7f43140a9750 legacy=0x7f43141469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 1 monclient: found mon.noname-b 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43141470b0 con 0x7f4314149890 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4310001a10 con 0x7f4314149890 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.508+0000 7f4318acc640 10 monclient: dump: 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:05.498 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:05.499 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.510+0000 7f431bd32640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:28:05.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.510+0000 7f431bd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:05.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4310001be0 con 0x7f4314149890 2024-03-29T11:28:05.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f4318acc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f4318acc640 1 -- 172.21.15.52:0/3784367461 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4310001ff0 con 0x7f4314149890 2024-03-29T11:28:05.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f431bd32640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f431bd32640 4 set_mon_vals no callback set 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f4318acc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f4318acc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.511+0000 7f4318acc640 10 monclient: dump: 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:05.501 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:05.502 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.512+0000 7f431bd32640 10 monclient: shutdown 2024-03-29T11:28:05.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.512+0000 7f431bd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.512+0000 7f431bd32640 1 -- 172.21.15.52:0/3784367461 >> v1:172.21.15.52:6790/0 conn(0x7f4314149890 legacy=0x7f4314149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.514+0000 7f431bd32640 1 -- 172.21.15.52:0/3784367461 shutdown_connections 2024-03-29T11:28:05.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.514+0000 7f431bd32640 1 -- 172.21.15.52:0/3784367461 >> 172.21.15.52:0/3784367461 conn(0x7f43140b3a00 msgr2=0x7f43140b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.515+0000 7f431bd32640 1 -- 172.21.15.52:0/3784367461 shutdown_connections 2024-03-29T11:28:05.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.515+0000 7f431bd32640 1 -- 172.21.15.52:0/3784367461 wait complete. 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 10 monclient: monmap: 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 1 librados: starting msgr at 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 1 librados: starting objecter 2024-03-29T11:28:05.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 1 Processor -- start 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 1 -- start start 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 1 librados: setting wanted keys 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 1 librados: calling monclient init 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.516+0000 7f431bd32640 10 monclient: init 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient(hunting): picked mon.noname-c con 0x7f43140a9750 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient(hunting): picked mon.noname-b con 0x7f43140afa30 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient(hunting): picked mon.noname-a con 0x7f4314149890 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4314070700 con 0x7f4314149890 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43141dea10 con 0x7f43140afa30 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431406f500 con 0x7f43140a9750 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.517+0000 7f431bd32640 10 monclient(hunting): authenticate will time out at 1415.657813s 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.518+0000 7f4319ace640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f43140a9750 0x7f43141def30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49626/0 (socket says 172.21.15.52:49626) 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.519+0000 7f4319ace640 1 -- 172.21.15.52:0/1253313733 learned_addr learned my addr 172.21.15.52:0/1253313733 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.519+0000 7f430a7fc640 10 client.?.objecter ms_handle_connect 0x7f43140a9750 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.519+0000 7f430a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.520+0000 7f430a7fc640 10 client.?.objecter ms_handle_connect 0x7f43140afa30 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538869171 0 0) 0x7f43141dea10 con 0x7f43140afa30 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:28:05.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f8004eb0 con 0x7f43140afa30 2024-03-29T11:28:05.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.521+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461913031 0 0) 0x7f431406f500 con 0x7f43140a9750 2024-03-29T11:28:05.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43141dea10 con 0x7f43140a9750 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226357206 0 0) 0x7f42f8004eb0 con 0x7f43140afa30 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f431406f500 con 0x7f43140afa30 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4310001ef0 con 0x7f43140afa30 2024-03-29T11:28:05.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.522+0000 7f430a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 10 monclient(hunting): dump: 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 377029351 0 0) 0x7f43141dea10 con 0x7f43140a9750 2024-03-29T11:28:05.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42f8004eb0 con 0x7f43140a9750 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f4304001990 con 0x7f43140a9750 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 10 monclient(hunting): dump: 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.523+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 670462922 0 0) 0x7f42f8004eb0 con 0x7f43140a9750 2024-03-29T11:28:05.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 >> v1:172.21.15.79:6789/0 conn(0x7f43140afa30 legacy=0x7f4314071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 >> v1:172.21.15.52:6789/0 conn(0x7f4314149890 legacy=0x7f43141e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 1 monclient: found mon.c 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431406e030 con 0x7f43140a9750 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.524+0000 7f430a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 10 monclient: _renew_subs 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4314144510 con 0x7f43140a9750 2024-03-29T11:28:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 10 monclient: _renew_subs 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f431406ea10 con 0x7f43140a9750 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.527+0000 7f431bd32640 1 librados: init done 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4304003610 con 0x7f43140a9750 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f43040039e0 con 0x7f43140a9750 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430a7fc640 10 monclient: dump: 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.520 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.528+0000 7f430bfff640 4 set_mon_vals no callback set 2024-03-29T11:28:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.530+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f4304003b40 con 0x7f43140a9750 2024-03-29T11:28:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.530+0000 7f431bd32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.530+0000 7f431bd32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.530+0000 7f431bd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.530+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f4304084a20 con 0x7f43140a9750 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f430a7fc640 10 client.4553.objecter ms_dispatch 0x7f4314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f430a7fc640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f430a7fc640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f430a7fc640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc005080 con 0x7f43140a9750 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f43192cd640 10 monclient: get_auth_request con 0x7f42f806de70 auth_method 0 2024-03-29T11:28:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.531+0000 7f430a7fc640 10 client.4553.objecter ms_handle_connect 0x7f42f806de70 2024-03-29T11:28:05.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.542+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4304050300 con 0x7f43140a9750 2024-03-29T11:28:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.542+0000 7f430a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.542+0000 7f430a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:05.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.880+0000 7f431bd32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.881+0000 7f431bd32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.881+0000 7f431bd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:05.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.881+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42fc005330 con 0x7f43140a9750 2024-03-29T11:28:05.872 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:05.872 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:05.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.882+0000 7f430a7fc640 1 -- 172.21.15.52:0/1253313733 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f4304052230 con 0x7f43140a9750 2024-03-29T11:28:05.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.882+0000 7f430a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.882+0000 7f430a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:05.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.886+0000 7f431bd32640 10 librados: watch_flush enter 2024-03-29T11:28:05.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.886+0000 7f431bd32640 10 librados: watch_flush exit 2024-03-29T11:28:05.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.886+0000 7f431bd32640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:28:05.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.886+0000 7f431bd32640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:28:05.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 >> v1:172.21.15.79:6816/407480584 conn(0x7f42f806de70 legacy=0x7f42f8070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 10 monclient: shutdown 2024-03-29T11:28:05.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 >> v1:172.21.15.52:6790/0 conn(0x7f43140a9750 legacy=0x7f43141def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 shutdown_connections 2024-03-29T11:28:05.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 >> 172.21.15.52:0/1253313733 conn(0x7f43140b3a00 msgr2=0x7f43140ac9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 shutdown_connections 2024-03-29T11:28:05.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 -- 172.21.15.52:0/1253313733 wait complete. 2024-03-29T11:28:05.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:05.888+0000 7f431bd32640 1 librados: shutdown 2024-03-29T11:28:05.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:12.704 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.865+0000 7f15a5e02640 10 monclient: get_monmap_and_config 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.865+0000 7f15a5e02640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.865+0000 7f15a5e02640 10 monclient: monmap: 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:12.866736+0000 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:12.866736+0000 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:12.855 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:12.856 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:12.856 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:12.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.867+0000 7f15a5e02640 1 Processor -- start 2024-03-29T11:28:12.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.868+0000 7f15a5e02640 1 -- start start 2024-03-29T11:28:12.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.868+0000 7f15a5e02640 10 monclient: init 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient(hunting): picked mon.noname-c con 0x7f15a0149a40 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient(hunting): picked mon.noname-a con 0x7f15a0071f80 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient(hunting): picked mon.noname-b con 0x7f15a00a9e00 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00aa2b0 con 0x7f15a0071f80 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00ab7a0 con 0x7f15a0149a40 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00aba80 con 0x7f15a00a9e00 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5e02640 10 monclient(hunting): authenticate will time out at 1423.009305s 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5601640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f15a00a9e00 0x7f15a0151b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:46720/0 (socket says 172.21.15.52:46720) 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.869+0000 7f15a5601640 1 -- 172.21.15.52:0/3305260644 learned_addr learned my addr 172.21.15.52:0/3305260644 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:12.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515206974 0 0) 0x7f15a00aba80 con 0x7f15a00a9e00 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588003520 con 0x7f15a00a9e00 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878911128 0 0) 0x7f1588003520 con 0x7f15a00a9e00 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 >> v1:172.21.15.79:6789/0 conn(0x7f15a0149a40 legacy=0x7f15a00ad470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 >> v1:172.21.15.52:6789/0 conn(0x7f15a0071f80 legacy=0x7f15a00a96f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.870+0000 7f159f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:28:12.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a014a240 con 0x7f15a00a9e00 2024-03-29T11:28:12.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f15a5e02640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f15a5e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f15900019f0 con 0x7f15a00a9e00 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: dump: 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.869 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f15a5e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1590001c50 con 0x7f15a00a9e00 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f159f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f15a5e02640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.871+0000 7f15a5e02640 4 set_mon_vals no callback set 2024-03-29T11:28:12.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f159f7fe640 1 -- 172.21.15.52:0/3305260644 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f15900019f0 con 0x7f15a00a9e00 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f159f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f159f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f159f7fe640 10 monclient: dump: 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:12.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f15a5e02640 10 monclient: shutdown 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f15a5e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.872+0000 7f15a5e02640 1 -- 172.21.15.52:0/3305260644 >> v1:172.21.15.52:6790/0 conn(0x7f15a00a9e00 legacy=0x7f15a0151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.873+0000 7f15a5e02640 1 -- 172.21.15.52:0/3305260644 shutdown_connections 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.873+0000 7f15a5e02640 1 -- 172.21.15.52:0/3305260644 >> 172.21.15.52:0/3305260644 conn(0x7f15a00b3ae0 msgr2=0x7f15a00b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.873+0000 7f15a5e02640 1 -- 172.21.15.52:0/3305260644 shutdown_connections 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.874+0000 7f15a5e02640 1 -- 172.21.15.52:0/3305260644 wait complete. 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.874+0000 7f15a5e02640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.874+0000 7f15a5e02640 10 monclient: monmap: 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:12.872 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 1 librados: starting msgr at 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 1 librados: starting objecter 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 1 Processor -- start 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 1 -- start start 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 1 librados: setting wanted keys 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 1 librados: calling monclient init 2024-03-29T11:28:12.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.875+0000 7f15a5e02640 10 monclient: init 2024-03-29T11:28:12.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.877+0000 7f15a5e02640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.877+0000 7f15a5e02640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:12.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.877+0000 7f15a5e02640 10 monclient(hunting): picked mon.noname-a con 0x7f15a0071f80 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:12.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.877+0000 7f15a5e02640 10 monclient(hunting): picked mon.noname-b con 0x7f15a00a9e00 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:12.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5e02640 10 monclient(hunting): picked mon.noname-c con 0x7f15a0149a40 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:12.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5e02640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a0145550 con 0x7f15a0071f80 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5e02640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a01443f0 con 0x7f15a00a9e00 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5e02640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a0146f00 con 0x7f15a0149a40 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5e02640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5e02640 10 monclient(hunting): authenticate will time out at 1423.018429s 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5601640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f15a0149a40 0x7f15a01db990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49000/0 (socket says 172.21.15.52:49000) 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f15a5601640 1 -- 172.21.15.52:0/579194624 learned_addr learned my addr 172.21.15.52:0/579194624 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f159d7fa640 10 client.?.objecter ms_handle_connect 0x7f15a0149a40 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f159d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354736350 0 0) 0x7f15a0146f00 con 0x7f15a0149a40 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f159d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f159d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.878+0000 7f159d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.879+0000 7f159d7fa640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.879+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158c004eb0 con 0x7f15a0149a40 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3170186460 0 0) 0x7f158c004eb0 con 0x7f15a0149a40 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a0146f00 con 0x7f15a0149a40 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1590001810 con 0x7f15a0149a40 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:12.879 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.881+0000 7f159d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.883+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1735364672 0 0) 0x7f15a0146f00 con 0x7f15a0149a40 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.883+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 >> v1:172.21.15.79:6789/0 conn(0x7f15a00a9e00 legacy=0x7f15a01db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.883+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 >> v1:172.21.15.52:6789/0 conn(0x7f15a0071f80 legacy=0x7f15a0144e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 1 monclient: found mon.c 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a0147090 con 0x7f15a0149a40 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1590001970 con 0x7f15a0149a40 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1590005310 con 0x7f15a0149a40 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.884+0000 7f159d7fa640 10 monclient: dump: 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:12.881 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 10 monclient: _renew_subs 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a0147280 con 0x7f15a0149a40 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 10 monclient: _renew_subs 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a01e6db0 con 0x7f15a0149a40 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f15a5e02640 1 librados: init done 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.886+0000 7f1586ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.885+0000 7f159effd640 4 set_mon_vals no callback set 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.888+0000 7f1586ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.888+0000 7f1586ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:12.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.888+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578005080 con 0x7f15a0149a40 2024-03-29T11:28:12.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.888+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f159001b2e0 con 0x7f15a0149a40 2024-03-29T11:28:12.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.889+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f15900851c0 con 0x7f15a0149a40 2024-03-29T11:28:12.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.889+0000 7f159d7fa640 10 client.4559.objecter ms_dispatch 0x7f15a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:12.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.889+0000 7f159d7fa640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:12.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.889+0000 7f159d7fa640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:12.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.889+0000 7f159d7fa640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:28:12.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.890+0000 7f15a4e00640 10 monclient: get_auth_request con 0x7f158c06dc50 auth_method 0 2024-03-29T11:28:12.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.893+0000 7f159d7fa640 10 client.4559.objecter ms_handle_connect 0x7f158c06dc50 2024-03-29T11:28:12.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.896+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1590050aa0 con 0x7f15a0149a40 2024-03-29T11:28:12.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.896+0000 7f159d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:12.896+0000 7f159d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:13.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.227+0000 7f1586ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.227+0000 7f1586ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.227+0000 7f1586ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:13.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.227+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1578005330 con 0x7f15a0149a40 2024-03-29T11:28:13.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.229+0000 7f159d7fa640 1 -- 172.21.15.52:0/579194624 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f15900529d0 con 0x7f15a0149a40 2024-03-29T11:28:13.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.229+0000 7f159d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.229+0000 7f159d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:13.218 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:13.218 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 10 librados: watch_flush enter 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 10 librados: watch_flush exit 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 >> v1:172.21.15.79:6816/407480584 conn(0x7f158c06dc50 legacy=0x7f158c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 10 monclient: shutdown 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 >> v1:172.21.15.52:6790/0 conn(0x7f15a0149a40 legacy=0x7f15a01db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 shutdown_connections 2024-03-29T11:28:13.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.232+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 >> 172.21.15.52:0/579194624 conn(0x7f15a00b3ae0 msgr2=0x7f15a00abf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:13.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.234+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 shutdown_connections 2024-03-29T11:28:13.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.234+0000 7f1586ffd640 1 -- 172.21.15.52:0/579194624 wait complete. 2024-03-29T11:28:13.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:13.234+0000 7f1586ffd640 1 librados: shutdown 2024-03-29T11:28:13.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:20.047 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.201+0000 7f92d359e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.201+0000 7f92d359e640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.201+0000 7f92d359e640 10 monclient: monmap: 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:20.202861+0000 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:20.202861+0000 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.204+0000 7f92d359e640 1 Processor -- start 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.204+0000 7f92d359e640 1 -- start start 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.204+0000 7f92d359e640 10 monclient: init 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.205+0000 7f92d359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.205+0000 7f92d359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.205+0000 7f92d359e640 10 monclient(hunting): picked mon.noname-b con 0x7f92d40a9750 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.205+0000 7f92d359e640 10 monclient(hunting): picked mon.noname-a con 0x7f92d40ae220 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.205+0000 7f92d359e640 10 monclient(hunting): picked mon.noname-c con 0x7f92d413b9d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.206+0000 7f92d359e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d4142490 con 0x7f92d40ae220 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.206+0000 7f92d359e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d40ac130 con 0x7f92d413b9d0 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.206+0000 7f92d359e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d40ac410 con 0x7f92d40a9750 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.206+0000 7f92d359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.206+0000 7f92d359e640 10 monclient(hunting): authenticate will time out at 1430.346342s 2024-03-29T11:28:20.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.211+0000 7f92d2d9d640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f92d413b9d0 0x7f92d4149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:46225/0 (socket says 172.21.15.52:46225) 2024-03-29T11:28:20.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.211+0000 7f92d2d9d640 1 -- 172.21.15.52:0/63785588 learned_addr learned my addr 172.21.15.52:0/63785588 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:20.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.212+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792738546 0 0) 0x7f92d40ac410 con 0x7f92d40a9750 2024-03-29T11:28:20.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.212+0000 7f92d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.212+0000 7f92d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.212+0000 7f92d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.212+0000 7f92d159a640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:28:20.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.212+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92bc003520 con 0x7f92d40a9750 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891457872 0 0) 0x7f92d4142490 con 0x7f92d40ae220 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d40ac410 con 0x7f92d40ae220 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905703243 0 0) 0x7f92d40ac130 con 0x7f92d413b9d0 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4142490 con 0x7f92d413b9d0 2024-03-29T11:28:20.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711024314 0 0) 0x7f92d40ac410 con 0x7f92d40ae220 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 >> v1:172.21.15.52:6790/0 conn(0x7f92d40a9750 legacy=0x7f92d4141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 >> v1:172.21.15.79:6789/0 conn(0x7f92d413b9d0 legacy=0x7f92d4149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 monclient: found mon.noname-a 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40afa40 con 0x7f92d40ae220 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f92c8001670 con 0x7f92d40ae220 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.213+0000 7f92d159a640 10 monclient: dump: 2024-03-29T11:28:20.207 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.214+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92c80017d0 con 0x7f92d40ae220 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.214+0000 7f92d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.214+0000 7f92d159a640 1 -- 172.21.15.52:0/63785588 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f92c8001ba0 con 0x7f92d40ae220 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.214+0000 7f92d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.214+0000 7f92d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.214+0000 7f92d159a640 10 monclient: dump: 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:20.209 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:20.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.216+0000 7f92d359e640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:28:20.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.216+0000 7f92d359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:20.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.216+0000 7f92d359e640 4 set_mon_vals no callback set 2024-03-29T11:28:20.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.217+0000 7f92d359e640 10 monclient: shutdown 2024-03-29T11:28:20.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.217+0000 7f92d359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.217+0000 7f92d359e640 1 -- 172.21.15.52:0/63785588 >> v1:172.21.15.52:6789/0 conn(0x7f92d40ae220 legacy=0x7f92d413b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.220+0000 7f92d359e640 1 -- 172.21.15.52:0/63785588 shutdown_connections 2024-03-29T11:28:20.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.220+0000 7f92d359e640 1 -- 172.21.15.52:0/63785588 >> 172.21.15.52:0/63785588 conn(0x7f92d40673e0 msgr2=0x7f92d40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.220+0000 7f92d359e640 1 -- 172.21.15.52:0/63785588 shutdown_connections 2024-03-29T11:28:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.221+0000 7f92d359e640 1 -- 172.21.15.52:0/63785588 wait complete. 2024-03-29T11:28:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.222+0000 7f92d359e640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.222+0000 7f92d359e640 10 monclient: monmap: 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.223+0000 7f92d359e640 1 librados: starting msgr at 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.223+0000 7f92d359e640 1 librados: starting objecter 2024-03-29T11:28:20.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.223+0000 7f92d359e640 1 Processor -- start 2024-03-29T11:28:20.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.225+0000 7f92d359e640 1 -- start start 2024-03-29T11:28:20.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.225+0000 7f92d359e640 1 librados: setting wanted keys 2024-03-29T11:28:20.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.225+0000 7f92d359e640 1 librados: calling monclient init 2024-03-29T11:28:20.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.225+0000 7f92d359e640 10 monclient: init 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.225+0000 7f92d359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.225+0000 7f92d359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 10 monclient(hunting): picked mon.noname-a con 0x7f92d40a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 10 monclient(hunting): picked mon.noname-c con 0x7f92d40ae220 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 10 monclient(hunting): picked mon.noname-b con 0x7f92d413b9d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d41404d0 con 0x7f92d40a9750 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d413c4f0 con 0x7f92d413b9d0 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d413be30 con 0x7f92d40ae220 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.226+0000 7f92d359e640 10 monclient(hunting): authenticate will time out at 1430.366142s 2024-03-29T11:28:20.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92d259c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f92d40a9750 0x7f92d41dbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:41929/0 (socket says 172.21.15.52:41929) 2024-03-29T11:28:20.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92d259c640 1 -- 172.21.15.52:0/100350822 learned_addr learned my addr 172.21.15.52:0/100350822 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:20.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 client.?.objecter ms_handle_connect 0x7f92d40a9750 2024-03-29T11:28:20.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817287293 0 0) 0x7f92d41404d0 con 0x7f92d40a9750 2024-03-29T11:28:20.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:28:20.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b4004eb0 con 0x7f92d40a9750 2024-03-29T11:28:20.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.231+0000 7f92c2ffd640 10 client.?.objecter ms_handle_connect 0x7f92d40ae220 2024-03-29T11:28:20.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 client.?.objecter ms_handle_connect 0x7f92d413b9d0 2024-03-29T11:28:20.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202935592 0 0) 0x7f92b4004eb0 con 0x7f92d40a9750 2024-03-29T11:28:20.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d41404d0 con 0x7f92d40a9750 2024-03-29T11:28:20.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206911387 0 0) 0x7f92d413be30 con 0x7f92d40ae220 2024-03-29T11:28:20.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b4004eb0 con 0x7f92d40ae220 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635963336 0 0) 0x7f92d413c4f0 con 0x7f92d413b9d0 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d413be30 con 0x7f92d413b9d0 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f92cc001ce0 con 0x7f92d40a9750 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:20.227 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3793109193 0 0) 0x7f92d41404d0 con 0x7f92d40a9750 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 >> v1:172.21.15.52:6790/0 conn(0x7f92d40ae220 legacy=0x7f92d41dedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 >> v1:172.21.15.79:6789/0 conn(0x7f92d413b9d0 legacy=0x7f92d41e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 monclient: found mon.a 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d41400d0 con 0x7f92d40a9750 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.232+0000 7f92c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92cc001e40 con 0x7f92d40a9750 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f92cc003de0 con 0x7f92d40a9750 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92d0d99640 4 set_mon_vals no callback set 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.233+0000 7f92c2ffd640 10 monclient: dump: 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:20.229 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:20.230 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 10 monclient: _renew_subs 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d41dc2f0 con 0x7f92d40a9750 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 10 monclient: _renew_subs 2024-03-29T11:28:20.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d413f7a0 con 0x7f92d40a9750 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.234+0000 7f92d359e640 1 librados: init done 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.235+0000 7f92d359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.236+0000 7f92d359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.236+0000 7f92d359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.236+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d404f730 con 0x7f92d40a9750 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.238+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f92cc003f40 con 0x7f92d40a9750 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.239+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f92cc084e30 con 0x7f92d40a9750 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.239+0000 7f92c2ffd640 10 client.4506.objecter ms_dispatch 0x7f92d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.239+0000 7f92c2ffd640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.239+0000 7f92c2ffd640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.239+0000 7f92d1d9b640 10 monclient: get_auth_request con 0x7f92b406dd90 auth_method 0 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.240+0000 7f92c2ffd640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T11:28:20.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.240+0000 7f92c2ffd640 10 client.4506.objecter ms_handle_connect 0x7f92b406dd90 2024-03-29T11:28:20.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.247+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92cc050540 con 0x7f92d40a9750 2024-03-29T11:28:20.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.247+0000 7f92c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.247+0000 7f92c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:20.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.585+0000 7f92d359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.585+0000 7f92d359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.585+0000 7f92d359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:20.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.585+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92d41402e0 con 0x7f92d40a9750 2024-03-29T11:28:20.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.587+0000 7f92c2ffd640 1 -- 172.21.15.52:0/100350822 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f92cc052640 con 0x7f92d40a9750 2024-03-29T11:28:20.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.587+0000 7f92c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.587+0000 7f92c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:20.578 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:20.578 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:20.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.591+0000 7f92d359e640 10 librados: watch_flush enter 2024-03-29T11:28:20.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.591+0000 7f92d359e640 10 librados: watch_flush exit 2024-03-29T11:28:20.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.591+0000 7f92d359e640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T11:28:20.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.591+0000 7f92d359e640 10 client.4506.objecter successfully canceled tick 2024-03-29T11:28:20.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.593+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 >> v1:172.21.15.79:6816/407480584 conn(0x7f92b406dd90 legacy=0x7f92b4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.593+0000 7f92d359e640 10 monclient: shutdown 2024-03-29T11:28:20.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.593+0000 7f92d359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.593+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 >> v1:172.21.15.52:6789/0 conn(0x7f92d40a9750 legacy=0x7f92d41dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.593+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 shutdown_connections 2024-03-29T11:28:20.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.593+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 >> 172.21.15.52:0/100350822 conn(0x7f92d40673e0 msgr2=0x7f92d4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.594+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 shutdown_connections 2024-03-29T11:28:20.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.594+0000 7f92d359e640 1 -- 172.21.15.52:0/100350822 wait complete. 2024-03-29T11:28:20.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:20.594+0000 7f92d359e640 1 librados: shutdown 2024-03-29T11:28:20.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:27.412 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:27.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.568+0000 7fdee5e0d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.568+0000 7fdee5e0d640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.568+0000 7fdee5e0d640 10 monclient: monmap: 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:27.569771+0000 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:27.569771+0000 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:27.559 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.569+0000 7fdee5e0d640 1 Processor -- start 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.570+0000 7fdee5e0d640 1 -- start start 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.570+0000 7fdee5e0d640 10 monclient: init 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.570+0000 7fdee5e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 10 monclient(hunting): picked mon.noname-c con 0x7fdee00a9ac0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 10 monclient(hunting): picked mon.noname-b con 0x7fdee013ee90 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 10 monclient(hunting): picked mon.noname-a con 0x7fdee00af900 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00afd10 con 0x7fdee00af900 2024-03-29T11:28:27.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00af6f0 con 0x7fdee00a9ac0 2024-03-29T11:28:27.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00aac80 con 0x7fdee013ee90 2024-03-29T11:28:27.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.571+0000 7fdee5e0d640 10 monclient(hunting): authenticate will time out at 1437.711657s 2024-03-29T11:28:27.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdedffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fdee00af900 0x7fdee0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:45802/0 (socket says 172.21.15.52:45802) 2024-03-29T11:28:27.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdedffff640 1 -- 172.21.15.52:0/2526084712 learned_addr learned my addr 172.21.15.52:0/2526084712 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261237445 0 0) 0x7fdee00afd10 con 0x7fdee00af900 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdede7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdede7fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.573+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc003520 con 0x7fdee00af900 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431794892 0 0) 0x7fdee00af6f0 con 0x7fdee00a9ac0 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:28:27.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee00afd10 con 0x7fdee00a9ac0 2024-03-29T11:28:27.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020769023 0 0) 0x7fdecc003520 con 0x7fdee00af900 2024-03-29T11:28:27.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.574+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 >> v1:172.21.15.52:6790/0 conn(0x7fdee013ee90 legacy=0x7fdee013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:27.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 >> v1:172.21.15.79:6789/0 conn(0x7fdee00a9ac0 legacy=0x7fdee00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 1 monclient: found mon.noname-a 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00aaf60 con 0x7fdee00af900 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fded8001a10 con 0x7fdee00af900 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdede7fc640 10 monclient: dump: 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:27.568 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:27.569 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdee5e0d640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:28:27.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.577+0000 7fdee5e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:27.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.578+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fded8001b70 con 0x7fdee00af900 2024-03-29T11:28:27.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.578+0000 7fdede7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.578+0000 7fdede7fc640 1 -- 172.21.15.52:0/2526084712 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fded8001fc0 con 0x7fdee00af900 2024-03-29T11:28:27.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.578+0000 7fdede7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.578+0000 7fdede7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.578+0000 7fdede7fc640 10 monclient: dump: 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.579+0000 7fdee5e0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:27.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.579+0000 7fdee5e0d640 4 set_mon_vals no callback set 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.579+0000 7fdee5e0d640 10 monclient: shutdown 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.579+0000 7fdee5e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.579+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2526084712 >> v1:172.21.15.52:6789/0 conn(0x7fdee00af900 legacy=0x7fdee0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.581+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2526084712 shutdown_connections 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.581+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2526084712 >> 172.21.15.52:0/2526084712 conn(0x7fdee00b35e0 msgr2=0x7fdee00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.581+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2526084712 shutdown_connections 2024-03-29T11:28:27.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.581+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2526084712 wait complete. 2024-03-29T11:28:27.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.582+0000 7fdee5e0d640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.583+0000 7fdee5e0d640 10 monclient: monmap: 2024-03-29T11:28:27.573 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:27.573 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:27.573 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:27.573 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.583+0000 7fdee5e0d640 1 librados: starting msgr at 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.583+0000 7fdee5e0d640 1 librados: starting objecter 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.583+0000 7fdee5e0d640 1 Processor -- start 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.584+0000 7fdee5e0d640 1 -- start start 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.584+0000 7fdee5e0d640 1 librados: setting wanted keys 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.584+0000 7fdee5e0d640 1 librados: calling monclient init 2024-03-29T11:28:27.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.584+0000 7fdee5e0d640 10 monclient: init 2024-03-29T11:28:27.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.585+0000 7fdee5e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.585+0000 7fdee5e0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:27.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.585+0000 7fdee5e0d640 10 monclient(hunting): picked mon.noname-a con 0x7fdee00a9ac0 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:27.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.585+0000 7fdee5e0d640 10 monclient(hunting): picked mon.noname-b con 0x7fdee00af900 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:27.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.586+0000 7fdedf7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fdee00a9ac0 0x7fdee01d69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:49966/0 (socket says 172.21.15.52:49966) 2024-03-29T11:28:27.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdee5e0d640 10 monclient(hunting): picked mon.noname-c con 0x7fdee013ee90 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:27.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdee5e0d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee013ff70 con 0x7fdee00a9ac0 2024-03-29T11:28:27.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdee5e0d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee01d6000 con 0x7fdee00af900 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdee5e0d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee01d6190 con 0x7fdee013ee90 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdee5e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdee5e0d640 10 monclient(hunting): authenticate will time out at 1437.728406s 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.588+0000 7fdedf7fe640 1 -- 172.21.15.52:0/2510571140 learned_addr learned my addr 172.21.15.52:0/2510571140 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 client.?.objecter ms_handle_connect 0x7fdee00a9ac0 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 client.?.objecter ms_handle_connect 0x7fdee013ee90 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618059654 0 0) 0x7fdee013ff70 con 0x7fdee00a9ac0 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec0004eb0 con 0x7fdee00a9ac0 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294895645 0 0) 0x7fdee01d6190 con 0x7fdee013ee90 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee013ff70 con 0x7fdee013ee90 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893816103 0 0) 0x7fdec0004eb0 con 0x7fdee00a9ac0 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdee01d6190 con 0x7fdee00a9ac0 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fded0001b10 con 0x7fdee00a9ac0 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): dump: 2024-03-29T11:28:27.582 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825576916 0 0) 0x7fdee013ff70 con 0x7fdee013ee90 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdec0006a40 con 0x7fdee013ee90 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fded8001ce0 con 0x7fdee013ee90 2024-03-29T11:28:27.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.589+0000 7fded7fff640 10 monclient(hunting): dump: 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.590+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 986488670 0 0) 0x7fdec0006a40 con 0x7fdee013ee90 2024-03-29T11:28:27.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.590+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 >> v1:172.21.15.79:6789/0 conn(0x7fdee00af900 legacy=0x7fdee01d2b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:27.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 >> v1:172.21.15.52:6789/0 conn(0x7fdee00a9ac0 legacy=0x7fdee01d69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 1 monclient: found mon.c 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0134fe0 con 0x7fdee013ee90 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fdee5e0d640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fdee5e0d640 10 monclient: _renew_subs 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fded8004cc0 con 0x7fdee013ee90 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fdee5e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee0135230 con 0x7fdee013ee90 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fded8004fd0 con 0x7fdee013ee90 2024-03-29T11:28:27.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.593+0000 7fded7fff640 10 monclient: dump: 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:27.590 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.594+0000 7fdee5e0d640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.594+0000 7fdee5e0d640 10 monclient: _renew_subs 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.594+0000 7fdee5e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.594+0000 7fdee5e0d640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee00b2230 con 0x7fdee013ee90 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.594+0000 7fdee5e0d640 1 librados: init done 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.594+0000 7fdedd7fa640 4 set_mon_vals no callback set 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.595+0000 7fded5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.595+0000 7fded5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.595+0000 7fded5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.595+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb8005080 con 0x7fdee013ee90 2024-03-29T11:28:27.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.596+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fded801afa0 con 0x7fdee013ee90 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fded8084d40 con 0x7fdee013ee90 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fded7fff640 10 client.4568.objecter ms_dispatch 0x7fdee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fded7fff640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fded7fff640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fded7fff640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fdedf7fe640 10 monclient: get_auth_request con 0x7fdec006e1e0 auth_method 0 2024-03-29T11:28:27.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.597+0000 7fded7fff640 10 client.4568.objecter ms_handle_connect 0x7fdec006e1e0 2024-03-29T11:28:27.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.605+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fded8050680 con 0x7fdee013ee90 2024-03-29T11:28:27.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.605+0000 7fded7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.605+0000 7fded7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:27.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.938+0000 7fded5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.938+0000 7fded5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.938+0000 7fded5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:27.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.938+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb8005330 con 0x7fdee013ee90 2024-03-29T11:28:27.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.941+0000 7fded7fff640 1 -- 172.21.15.52:0/2510571140 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fded80525b0 con 0x7fdee013ee90 2024-03-29T11:28:27.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.941+0000 7fded7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.941+0000 7fded7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:27.931 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:27.932 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.945+0000 7fded5ffb640 10 librados: watch_flush enter 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.945+0000 7fded5ffb640 10 librados: watch_flush exit 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.945+0000 7fded5ffb640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.945+0000 7fded5ffb640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.945+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 >> v1:172.21.15.79:6816/407480584 conn(0x7fdec006e1e0 legacy=0x7fdec00706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.945+0000 7fded5ffb640 10 monclient: shutdown 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.946+0000 7fded5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.946+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 >> v1:172.21.15.52:6790/0 conn(0x7fdee013ee90 legacy=0x7fdee01348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.947+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 shutdown_connections 2024-03-29T11:28:27.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.947+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 >> 172.21.15.52:0/2510571140 conn(0x7fdee00b35e0 msgr2=0x7fdee013b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.947+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 shutdown_connections 2024-03-29T11:28:27.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.947+0000 7fded5ffb640 1 -- 172.21.15.52:0/2510571140 wait complete. 2024-03-29T11:28:27.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:27.947+0000 7fded5ffb640 1 librados: shutdown 2024-03-29T11:28:27.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:34.765 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:34.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.939+0000 7f316078e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.939+0000 7f316078e640 10 monclient: build_initial_monmap 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.939+0000 7f316078e640 10 monclient: monmap: 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:34.940751+0000 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:34.940751+0000 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:34.930 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.940+0000 7f316078e640 1 Processor -- start 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.940+0000 7f316078e640 1 -- start start 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.940+0000 7f316078e640 10 monclient: init 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient(hunting): picked mon.noname-a con 0x7f31580b4790 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient(hunting): picked mon.noname-c con 0x7f31580aea90 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient(hunting): picked mon.noname-b con 0x7f31580b39f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31580b0c60 con 0x7f31580b4790 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31580af020 con 0x7f31580aea90 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315813fbd0 con 0x7f31580b39f0 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.941+0000 7f316078e640 10 monclient(hunting): authenticate will time out at 1445.081944s 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315dd29640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f31580aea90 0x7f31580af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:54636/0 (socket says 172.21.15.52:54636) 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315dd29640 1 -- 172.21.15.52:0/4227582216 learned_addr learned my addr 172.21.15.52:0/4227582216 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176136369 0 0) 0x7f315813fbd0 con 0x7f31580b39f0 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:34.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3144003520 con 0x7f31580b39f0 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940413298 0 0) 0x7f31580af020 con 0x7f31580aea90 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.942+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f315813fbd0 con 0x7f31580aea90 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734301075 0 0) 0x7f31580b0c60 con 0x7f31580b4790 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31580af020 con 0x7f31580b4790 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609991897 0 0) 0x7f3144003520 con 0x7f31580b39f0 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 >> v1:172.21.15.79:6789/0 conn(0x7f31580aea90 legacy=0x7f31580af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 >> v1:172.21.15.52:6789/0 conn(0x7f31580b4790 legacy=0x7f31580b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: _finish_hunting 0 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 monclient: found mon.noname-b 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315813fd60 con 0x7f31580b39f0 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: _finish_auth 0 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: _check_auth_tickets 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3150001a10 con 0x7f31580b39f0 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f315d528640 10 monclient: dump: 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:34.946 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:34.947 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f316078e640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.943+0000 7f316078e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3150001b70 con 0x7f31580b39f0 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f315d528640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f315d528640 1 -- 172.21.15.52:0/4227582216 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3150001fc0 con 0x7f31580b39f0 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f315d528640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f315d528640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f315d528640 10 monclient: dump: 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:34.948 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f316078e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f316078e640 4 set_mon_vals no callback set 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f316078e640 10 monclient: shutdown 2024-03-29T11:28:34.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f316078e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:34.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.944+0000 7f316078e640 1 -- 172.21.15.52:0/4227582216 >> v1:172.21.15.52:6790/0 conn(0x7f31580b39f0 legacy=0x7f315813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.945+0000 7f316078e640 1 -- 172.21.15.52:0/4227582216 shutdown_connections 2024-03-29T11:28:34.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.945+0000 7f316078e640 1 -- 172.21.15.52:0/4227582216 >> 172.21.15.52:0/4227582216 conn(0x7f3158066a00 msgr2=0x7f3158066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.945+0000 7f316078e640 1 -- 172.21.15.52:0/4227582216 shutdown_connections 2024-03-29T11:28:34.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.945+0000 7f316078e640 1 -- 172.21.15.52:0/4227582216 wait complete. 2024-03-29T11:28:34.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.946+0000 7f316078e640 10 monclient: build_initial_monmap 2024-03-29T11:28:34.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.946+0000 7f316078e640 10 monclient: monmap: 2024-03-29T11:28:34.951 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:34.951 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:34.951 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:34.951 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.946+0000 7f316078e640 1 librados: starting msgr at 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.946+0000 7f316078e640 1 librados: starting objecter 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.946+0000 7f316078e640 1 Processor -- start 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 1 -- start start 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 1 librados: setting wanted keys 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 1 librados: calling monclient init 2024-03-29T11:28:34.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 10 monclient: init 2024-03-29T11:28:34.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:34.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:34.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 10 monclient(hunting): picked mon.noname-b con 0x7f31580b39f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:34.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 10 monclient(hunting): picked mon.noname-c con 0x7f31580b4790 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 10 monclient(hunting): picked mon.noname-a con 0x7f3158071e30 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.947+0000 7f316078e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3158134520 con 0x7f3158071e30 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f316078e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315806f7b0 con 0x7f31580b39f0 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f316078e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3158071730 con 0x7f31580b4790 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f316078e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f316078e640 10 monclient(hunting): authenticate will time out at 1445.088086s 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f315e52a640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f31580b39f0 0x7f31581df1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:55430/0 (socket says 172.21.15.52:55430) 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f315e52a640 1 -- 172.21.15.52:0/3015606636 learned_addr learned my addr 172.21.15.52:0/3015606636 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f314effd640 10 client.?.objecter ms_handle_connect 0x7f31580b39f0 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f314effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f314effd640 10 client.?.objecter ms_handle_connect 0x7f3158071e30 2024-03-29T11:28:34.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.948+0000 7f314effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 client.?.objecter ms_handle_connect 0x7f31580b4790 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284675197 0 0) 0x7f315806f7b0 con 0x7f31580b39f0 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f313c004eb0 con 0x7f31580b39f0 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659867220 0 0) 0x7f3158134520 con 0x7f3158071e30 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f315806f7b0 con 0x7f3158071e30 2024-03-29T11:28:34.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243220390 0 0) 0x7f3158071730 con 0x7f31580b4790 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.949+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158134520 con 0x7f31580b4790 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1789440810 0 0) 0x7f313c004eb0 con 0x7f31580b39f0 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3158071730 con 0x7f31580b39f0 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3154001a50 con 0x7f31580b39f0 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:34.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 10 monclient(hunting): dump: 2024-03-29T11:28:34.958 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:34.958 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:34.958 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.958 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.950+0000 7f314effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.951+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1325304299 0 0) 0x7f315806f7b0 con 0x7f3158071e30 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.951+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f313c004eb0 con 0x7f3158071e30 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.951+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1627679642 0 0) 0x7f3158071730 con 0x7f31580b39f0 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.951+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 >> v1:172.21.15.52:6790/0 conn(0x7f31580b4790 legacy=0x7f3158133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.951+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 >> v1:172.21.15.52:6789/0 conn(0x7f3158071e30 legacy=0x7f3158133e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 1 monclient: found mon.b 2024-03-29T11:28:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158071a10 con 0x7f31580b39f0 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 10 monclient: _renew_subs 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315806f3b0 con 0x7f31580b39f0 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 10 monclient: _renew_subs 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:34.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3158142090 con 0x7f31580b39f0 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f316078e640 1 librados: init done 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3154001c90 con 0x7f31580b39f0 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3154003a40 con 0x7f31580b39f0 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f314effd640 10 monclient: dump: 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.962 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:34.963 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:34.963 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:34.963 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:34.963 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:34.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.952+0000 7f315cd27640 4 set_mon_vals no callback set 2024-03-29T11:28:34.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.954+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f3154003ba0 con 0x7f31580b39f0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f316078e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f316078e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f316078e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3140005080 con 0x7f31580b39f0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f3154084ac0 con 0x7f31580b39f0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f315dd29640 10 monclient: get_auth_request con 0x7f313c06df30 auth_method 0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f314effd640 10 client.4501.objecter ms_dispatch 0x7f3158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f314effd640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f314effd640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.955+0000 7f314effd640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.956+0000 7f314effd640 10 client.4501.objecter ms_handle_connect 0x7f313c06df30 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.963+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31540503a0 con 0x7f31580b39f0 2024-03-29T11:28:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.963+0000 7f314effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:34.963+0000 7f314effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:35.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.324+0000 7f316078e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.324+0000 7f316078e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.324+0000 7f316078e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:35.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.324+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3140005330 con 0x7f31580b39f0 2024-03-29T11:28:35.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.326+0000 7f314effd640 1 -- 172.21.15.52:0/3015606636 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f31540522d0 con 0x7f31580b39f0 2024-03-29T11:28:35.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.326+0000 7f314effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.326+0000 7f314effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:35.317 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:35.317 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.329+0000 7f316078e640 10 librados: watch_flush enter 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.329+0000 7f316078e640 10 librados: watch_flush exit 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.329+0000 7f316078e640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.329+0000 7f316078e640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.330+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 >> v1:172.21.15.79:6816/407480584 conn(0x7f313c06df30 legacy=0x7f313c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.330+0000 7f316078e640 10 monclient: shutdown 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.330+0000 7f316078e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:35.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.330+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 >> v1:172.21.15.79:6789/0 conn(0x7f31580b39f0 legacy=0x7f31581df1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.330+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 shutdown_connections 2024-03-29T11:28:35.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.330+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 >> 172.21.15.52:0/3015606636 conn(0x7f3158066a00 msgr2=0x7f31580b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.333+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 shutdown_connections 2024-03-29T11:28:35.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.333+0000 7f316078e640 1 -- 172.21.15.52:0/3015606636 wait complete. 2024-03-29T11:28:35.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:35.333+0000 7f316078e640 1 librados: shutdown 2024-03-29T11:28:35.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:42.145 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:42.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.302+0000 7feb4577b640 10 monclient: get_monmap_and_config 2024-03-29T11:28:42.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.302+0000 7feb4577b640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.302+0000 7feb4577b640 10 monclient: monmap: 2024-03-29T11:28:42.294 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:42.304751+0000 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:42.304751+0000 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.303+0000 7feb4577b640 1 Processor -- start 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.303+0000 7feb4577b640 1 -- start start 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.303+0000 7feb4577b640 10 monclient: init 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient(hunting): picked mon.noname-a con 0x7feb401457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient(hunting): picked mon.noname-b con 0x7feb4013f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient(hunting): picked mon.noname-c con 0x7feb400a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400aaef0 con 0x7feb401457f0 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb40145bd0 con 0x7feb400a9d70 2024-03-29T11:28:42.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb40145eb0 con 0x7feb4013f650 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.304+0000 7feb4577b640 10 monclient(hunting): authenticate will time out at 1452.445972s 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb367fc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7feb4013f650 0x7feb400b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41378/0 (socket says 172.21.15.52:41378) 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb367fc640 1 -- 172.21.15.52:0/2840981769 learned_addr learned my addr 172.21.15.52:0/2840981769 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235746040 0 0) 0x7feb40145eb0 con 0x7feb4013f650 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb3e7fc640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.305+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb2c003520 con 0x7feb4013f650 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3292793870 0 0) 0x7feb2c003520 con 0x7feb4013f650 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 >> v1:172.21.15.79:6789/0 conn(0x7feb400a9d70 legacy=0x7feb4014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:42.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 >> v1:172.21.15.52:6789/0 conn(0x7feb401457f0 legacy=0x7feb4013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40146340 con 0x7feb4013f650 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7feb28001690 con 0x7feb4013f650 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: dump: 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:42.310 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb4577b640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb4577b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb28001860 con 0x7feb4013f650 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 1 -- 172.21.15.52:0/2840981769 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7feb28001c70 con 0x7feb4013f650 2024-03-29T11:28:42.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb3e7fc640 10 monclient: dump: 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb4577b640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.306+0000 7feb4577b640 4 set_mon_vals no callback set 2024-03-29T11:28:42.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 10 monclient: shutdown 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 1 -- 172.21.15.52:0/2840981769 >> v1:172.21.15.52:6790/0 conn(0x7feb4013f650 legacy=0x7feb400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 1 -- 172.21.15.52:0/2840981769 shutdown_connections 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 1 -- 172.21.15.52:0/2840981769 >> 172.21.15.52:0/2840981769 conn(0x7feb40066a00 msgr2=0x7feb40066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 1 -- 172.21.15.52:0/2840981769 shutdown_connections 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.307+0000 7feb4577b640 1 -- 172.21.15.52:0/2840981769 wait complete. 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.308+0000 7feb4577b640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.308+0000 7feb4577b640 10 monclient: monmap: 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.313 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.308+0000 7feb4577b640 1 librados: starting msgr at 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.308+0000 7feb4577b640 1 librados: starting objecter 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.309+0000 7feb4577b640 1 Processor -- start 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.309+0000 7feb4577b640 1 -- start start 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.309+0000 7feb4577b640 1 librados: setting wanted keys 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.309+0000 7feb4577b640 1 librados: calling monclient init 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.309+0000 7feb4577b640 10 monclient: init 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb4577b640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb4577b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb4577b640 10 monclient(hunting): picked mon.noname-c con 0x7feb400a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb3effd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7feb400a9d70 0x7feb401dab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41534/0 (socket says 172.21.15.52:41534) 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb3effd640 1 -- 172.21.15.52:0/819970528 learned_addr learned my addr 172.21.15.52:0/819970528 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb4577b640 10 monclient(hunting): picked mon.noname-a con 0x7feb4013f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb37fff640 10 client.?.objecter ms_handle_connect 0x7feb4013f650 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb37fff640 10 client.?.objecter ms_handle_connect 0x7feb400a9d70 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.310+0000 7feb37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb4577b640 10 monclient(hunting): picked mon.noname-b con 0x7feb401457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400af580 con 0x7feb4013f650 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb401db290 con 0x7feb401457f0 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400b02c0 con 0x7feb400a9d70 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb4577b640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb4577b640 10 monclient(hunting): authenticate will time out at 1452.452494s 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926312545 0 0) 0x7feb400b02c0 con 0x7feb400a9d70 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20004eb0 con 0x7feb400a9d70 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.311+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338300937 0 0) 0x7feb400af580 con 0x7feb4013f650 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb400b02c0 con 0x7feb4013f650 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 client.?.objecter ms_handle_connect 0x7feb401457f0 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3558295276 0 0) 0x7feb20004eb0 con 0x7feb400a9d70 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb400af580 con 0x7feb400a9d70 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7feb30001d10 con 0x7feb400a9d70 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): dump: 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:42.317 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4266625277 0 0) 0x7feb400b02c0 con 0x7feb4013f650 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb20004eb0 con 0x7feb4013f650 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7feb28001b70 con 0x7feb4013f650 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.312+0000 7feb37fff640 10 monclient(hunting): dump: 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.318 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226292700 0 0) 0x7feb401db290 con 0x7feb401457f0 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb400b02c0 con 0x7feb401457f0 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 858256948 0 0) 0x7feb400af580 con 0x7feb400a9d70 2024-03-29T11:28:42.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 >> v1:172.21.15.79:6789/0 conn(0x7feb401457f0 legacy=0x7feb401e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 >> v1:172.21.15.52:6789/0 conn(0x7feb4013f650 legacy=0x7feb40139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 monclient: found mon.c 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb400b0de0 con 0x7feb400a9d70 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb4577b640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb4577b640 10 monclient: _renew_subs 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb4577b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb400b1030 con 0x7feb400a9d70 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.313+0000 7feb4577b640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:42.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb4577b640 10 monclient: _renew_subs 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb4577b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb40142620 con 0x7feb400a9d70 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb4577b640 1 librados: init done 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb30001e70 con 0x7feb400a9d70 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7feb30003de0 con 0x7feb400a9d70 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb37fff640 10 monclient: dump: 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:42.321 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.314+0000 7feb3dffb640 4 set_mon_vals no callback set 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.315+0000 7feb4577b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.315+0000 7feb4577b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.315+0000 7feb4577b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.315+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb24005080 con 0x7feb400a9d70 2024-03-29T11:28:42.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.317+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7feb30003f40 con 0x7feb400a9d70 2024-03-29T11:28:42.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.317+0000 7feb367fc640 10 monclient: get_auth_request con 0x7feb2006dfe0 auth_method 0 2024-03-29T11:28:42.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.317+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7feb30084e00 con 0x7feb400a9d70 2024-03-29T11:28:42.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.317+0000 7feb37fff640 10 client.4580.objecter ms_dispatch 0x7feb40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:42.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.317+0000 7feb37fff640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.317+0000 7feb37fff640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.318+0000 7feb37fff640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:28:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.318+0000 7feb37fff640 10 client.4580.objecter ms_handle_connect 0x7feb2006dfe0 2024-03-29T11:28:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.327+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb30050510 con 0x7feb400a9d70 2024-03-29T11:28:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.327+0000 7feb37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.327+0000 7feb37fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:42.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.714+0000 7feb4577b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.714+0000 7feb4577b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.714+0000 7feb4577b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:28:42.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.714+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb24005330 con 0x7feb400a9d70 2024-03-29T11:28:42.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.716+0000 7feb37fff640 1 -- 172.21.15.52:0/819970528 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7feb30052610 con 0x7feb400a9d70 2024-03-29T11:28:42.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.716+0000 7feb37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.716+0000 7feb37fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:42.707 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:42.707 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 10 librados: watch_flush enter 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 10 librados: watch_flush exit 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 >> v1:172.21.15.79:6816/407480584 conn(0x7feb2006dfe0 legacy=0x7feb200704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 10 monclient: shutdown 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 >> v1:172.21.15.52:6790/0 conn(0x7feb400a9d70 legacy=0x7feb401dab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 shutdown_connections 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.719+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 >> 172.21.15.52:0/819970528 conn(0x7feb40066a00 msgr2=0x7feb400b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.720+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 shutdown_connections 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.720+0000 7feb4577b640 1 -- 172.21.15.52:0/819970528 wait complete. 2024-03-29T11:28:42.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:42.720+0000 7feb4577b640 1 librados: shutdown 2024-03-29T11:28:42.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:49.534 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.682+0000 7f2323fa8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.682+0000 7f2323fa8640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.682+0000 7f2323fa8640 10 monclient: monmap: 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:49.683561+0000 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:49.683561+0000 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.673 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.674 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:49.674 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:49.674 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:49.674 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.683+0000 7f2323fa8640 1 Processor -- start 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.683+0000 7f2323fa8640 1 -- start start 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.683+0000 7f2323fa8640 10 monclient: init 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f231c1422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f231c142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f231c0b17d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c142c80 con 0x7f231c142690 2024-03-29T11:28:49.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0aae40 con 0x7f231c1422b0 2024-03-29T11:28:49.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c13fbd0 con 0x7f231c0b17d0 2024-03-29T11:28:49.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.684+0000 7f2323fa8640 10 monclient(hunting): authenticate will time out at 1459.824796s 2024-03-29T11:28:49.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2322545640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f231c0b17d0 0x7f231c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42866/0 (socket says 172.21.15.52:42866) 2024-03-29T11:28:49.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2322545640 1 -- 172.21.15.52:0/1696356489 learned_addr learned my addr 172.21.15.52:0/1696356489 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:49.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148175921 0 0) 0x7f231c13fbd0 con 0x7f231c0b17d0 2024-03-29T11:28:49.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308003520 con 0x7f231c0b17d0 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778559773 0 0) 0x7f2308003520 con 0x7f231c0b17d0 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 >> v1:172.21.15.79:6789/0 conn(0x7f231c1422b0 legacy=0x7f231c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.685+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 >> v1:172.21.15.52:6789/0 conn(0x7f231c142690 legacy=0x7f231c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 1 monclient: found mon.noname-b 2024-03-29T11:28:49.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c151940 con 0x7f231c0b17d0 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2314001670 con 0x7f231c0b17d0 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2320d42640 10 monclient: dump: 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.683 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2323fa8640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.686+0000 7f2323fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23140018d0 con 0x7f231c0b17d0 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2320d42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2320d42640 1 -- 172.21.15.52:0/1696356489 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2314001ca0 con 0x7f231c0b17d0 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2323fa8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2323fa8640 4 set_mon_vals no callback set 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2320d42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2320d42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2320d42640 10 monclient: dump: 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2323fa8640 10 monclient: shutdown 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2323fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2323fa8640 1 -- 172.21.15.52:0/1696356489 >> v1:172.21.15.52:6790/0 conn(0x7f231c0b17d0 legacy=0x7f231c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.687+0000 7f2323fa8640 1 -- 172.21.15.52:0/1696356489 shutdown_connections 2024-03-29T11:28:49.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.688+0000 7f2323fa8640 1 -- 172.21.15.52:0/1696356489 >> 172.21.15.52:0/1696356489 conn(0x7f231c066a00 msgr2=0x7f231c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.688+0000 7f2323fa8640 1 -- 172.21.15.52:0/1696356489 shutdown_connections 2024-03-29T11:28:49.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.688+0000 7f2323fa8640 1 -- 172.21.15.52:0/1696356489 wait complete. 2024-03-29T11:28:49.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.689+0000 7f2323fa8640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.689+0000 7f2323fa8640 10 monclient: monmap: 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 1 librados: starting msgr at 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 1 librados: starting objecter 2024-03-29T11:28:49.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 1 Processor -- start 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 1 -- start start 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 1 librados: setting wanted keys 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 1 librados: calling monclient init 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.690+0000 7f2323fa8640 10 monclient: init 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.691+0000 7f2323fa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.691+0000 7f2323fa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.691+0000 7f2323fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f231c0b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:49.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.691+0000 7f2323fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f231c1422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.691+0000 7f2321d44640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f231c0b17d0 0x7f231c0b0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:50944/0 (socket says 172.21.15.52:50944) 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.691+0000 7f2321d44640 1 -- 172.21.15.52:0/4069155971 learned_addr learned my addr 172.21.15.52:0/4069155971 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.692+0000 7f23127fc640 10 client.?.objecter ms_handle_connect 0x7f231c0b17d0 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.692+0000 7f23127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f23127fc640 10 client.?.objecter ms_handle_connect 0x7f231c1422b0 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f23127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f2323fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f231c142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0acc80 con 0x7f231c142690 2024-03-29T11:28:49.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0acf60 con 0x7f231c0b17d0 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0ad240 con 0x7f231c1422b0 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f2323fa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.693+0000 7f2323fa8640 10 monclient(hunting): authenticate will time out at 1459.833585s 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 client.?.objecter ms_handle_connect 0x7f231c142690 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685314328 0 0) 0x7f231c0acc80 con 0x7f231c142690 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2300004eb0 con 0x7f231c142690 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989980435 0 0) 0x7f231c0ad240 con 0x7f231c1422b0 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.694+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231c0acc80 con 0x7f231c1422b0 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 345941479 0 0) 0x7f2300004eb0 con 0x7f231c142690 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f231c0ad240 con 0x7f231c142690 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2314001be0 con 0x7f231c142690 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): dump: 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.695 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305203280 0 0) 0x7f231c0acc80 con 0x7f231c1422b0 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2300004eb0 con 0x7f231c1422b0 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f230c001770 con 0x7f231c1422b0 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient(hunting): dump: 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.696 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 292645539 0 0) 0x7f231c0ad240 con 0x7f231c142690 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 >> v1:172.21.15.52:6790/0 conn(0x7f231c1422b0 legacy=0x7f231c1dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 >> v1:172.21.15.79:6789/0 conn(0x7f231c0b17d0 legacy=0x7f231c0b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.697 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 monclient: found mon.a 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c1329b0 con 0x7f231c142690 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.695+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2314001d40 con 0x7f231c142690 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f23127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f2313fff640 4 set_mon_vals no callback set 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2314004d90 con 0x7f231c142690 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f23127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f23127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:49.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f23127fc640 10 monclient: dump: 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f2323fa8640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f2323fa8640 10 monclient: _renew_subs 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f2323fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c0abfb0 con 0x7f231c142690 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.696+0000 7f2323fa8640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:49.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.697+0000 7f2323fa8640 10 monclient: _renew_subs 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.697+0000 7f2323fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.697+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c0abb40 con 0x7f231c142690 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.697+0000 7f2323fa8640 1 librados: init done 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.699+0000 7f2323fa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.699+0000 7f2323fa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.699+0000 7f2323fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.699+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c132810 con 0x7f231c142690 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.699+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f231401ad60 con 0x7f231c142690 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.700+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f2314084ab0 con 0x7f231c142690 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.700+0000 7f23127fc640 10 client.4527.objecter ms_dispatch 0x7f231c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.700+0000 7f23127fc640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.700+0000 7f23127fc640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.700+0000 7f23127fc640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T11:28:49.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.701+0000 7f2321d44640 10 monclient: get_auth_request con 0x7f230006df70 auth_method 0 2024-03-29T11:28:49.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.701+0000 7f23127fc640 10 client.4527.objecter ms_handle_connect 0x7f230006df70 2024-03-29T11:28:49.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.707+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f231c132810 con 0x7f231c142690 2024-03-29T11:28:49.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.707+0000 7f23127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:49.707+0000 7f23127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:50.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.039+0000 7f2323fa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.039+0000 7f2323fa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.039+0000 7f2323fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:28:50.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.039+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f231c132810 con 0x7f231c142690 2024-03-29T11:28:50.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.041+0000 7f23127fc640 1 -- 172.21.15.52:0/4069155971 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f231c132810 con 0x7f231c142690 2024-03-29T11:28:50.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.041+0000 7f23127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.041+0000 7f23127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:50.032 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:50.032 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:50.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.044+0000 7f2323fa8640 10 librados: watch_flush enter 2024-03-29T11:28:50.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 10 librados: watch_flush exit 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 10 client.4527.objecter successfully canceled tick 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 >> v1:172.21.15.79:6816/407480584 conn(0x7f230006df70 legacy=0x7f2300070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 10 monclient: shutdown 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.045+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 >> v1:172.21.15.52:6789/0 conn(0x7f231c142690 legacy=0x7f231c1db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.046+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 shutdown_connections 2024-03-29T11:28:50.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.046+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 >> 172.21.15.52:0/4069155971 conn(0x7f231c066a00 msgr2=0x7f231c13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.046+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 shutdown_connections 2024-03-29T11:28:50.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.046+0000 7f2323fa8640 1 -- 172.21.15.52:0/4069155971 wait complete. 2024-03-29T11:28:50.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:50.046+0000 7f2323fa8640 1 librados: shutdown 2024-03-29T11:28:50.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:56.861 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.007+0000 7f48ff0ff640 10 monclient: get_monmap_and_config 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.007+0000 7f48ff0ff640 10 monclient: build_initial_monmap 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.007+0000 7f48ff0ff640 10 monclient: monmap: 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:28:57.008434+0000 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:28:57.008434+0000 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:56.998 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:28:56.999 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:28:56.999 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:56.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.008+0000 7f48ff0ff640 1 Processor -- start 2024-03-29T11:28:56.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.008+0000 7f48ff0ff640 1 -- start start 2024-03-29T11:28:56.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.008+0000 7f48ff0ff640 10 monclient: init 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.009+0000 7f48ff0ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.009+0000 7f48ff0ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.009+0000 7f48ff0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f48f81457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.009+0000 7f48ff0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f48f813f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.009+0000 7f48fce9b640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f48f81457f0 0x7f48f813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:34478/0 (socket says 172.21.15.52:34478) 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48ff0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f48f80a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48ff0ff640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f80aaef0 con 0x7f48f80a9d70 2024-03-29T11:28:57.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48ff0ff640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f8145bd0 con 0x7f48f813f650 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48ff0ff640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f8145eb0 con 0x7f48f81457f0 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48ff0ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48ff0ff640 10 monclient(hunting): authenticate will time out at 1467.150167s 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48fce9b640 1 -- 172.21.15.52:0/4176540234 learned_addr learned my addr 172.21.15.52:0/4176540234 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550111295 0 0) 0x7f48f8145eb0 con 0x7f48f81457f0 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48f77fe640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:28:57.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.010+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e8003520 con 0x7f48f81457f0 2024-03-29T11:28:57.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.011+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1485871125 0 0) 0x7f48e8003520 con 0x7f48f81457f0 2024-03-29T11:28:57.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.011+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 >> v1:172.21.15.79:6789/0 conn(0x7f48f813f650 legacy=0x7f48f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:57.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.011+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 >> v1:172.21.15.52:6789/0 conn(0x7f48f80a9d70 legacy=0x7f48f814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 1 monclient: found mon.noname-b 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8146340 con 0x7f48f81457f0 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f48ec001690 con 0x7f48f81457f0 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:57.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48f77fe640 10 monclient: dump: 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:57.004 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:57.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48ff0ff640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.012+0000 7f48ff0ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48ec001860 con 0x7f48f81457f0 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48f77fe640 1 -- 172.21.15.52:0/4176540234 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f48ec001cb0 con 0x7f48f81457f0 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48f77fe640 10 monclient: dump: 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.007 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48ff0ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.013+0000 7f48ff0ff640 4 set_mon_vals no callback set 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 10 monclient: shutdown 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 1 -- 172.21.15.52:0/4176540234 >> v1:172.21.15.52:6790/0 conn(0x7f48f81457f0 legacy=0x7f48f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 1 -- 172.21.15.52:0/4176540234 shutdown_connections 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 1 -- 172.21.15.52:0/4176540234 >> 172.21.15.52:0/4176540234 conn(0x7f48f8066a00 msgr2=0x7f48f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 1 -- 172.21.15.52:0/4176540234 shutdown_connections 2024-03-29T11:28:57.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.014+0000 7f48ff0ff640 1 -- 172.21.15.52:0/4176540234 wait complete. 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.015+0000 7f48ff0ff640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.015+0000 7f48ff0ff640 10 monclient: monmap: 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 1 librados: starting msgr at 2024-03-29T11:28:57.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 1 librados: starting objecter 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 1 Processor -- start 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 1 -- start start 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 1 librados: setting wanted keys 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 1 librados: calling monclient init 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.016+0000 7f48ff0ff640 10 monclient: init 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48ff0ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48ff0ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:57.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48ff0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f48f80a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:28:57.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48fce9b640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f48f80a9d70 0x7f48f813b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37960/0 (socket says 172.21.15.52:37960) 2024-03-29T11:28:57.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48fce9b640 1 -- 172.21.15.52:0/321409978 learned_addr learned my addr 172.21.15.52:0/321409978 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:28:57.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48f57fa640 10 client.?.objecter ms_handle_connect 0x7f48f80a9d70 2024-03-29T11:28:57.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48f57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.017+0000 7f48ff0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f48f813f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48ff0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f48f81457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f813b970 con 0x7f48f81457f0 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f8139af0 con 0x7f48f813f650 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f8139c80 con 0x7f48f80a9d70 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48ff0ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48f57fa640 10 client.?.objecter ms_handle_connect 0x7f48f813f650 2024-03-29T11:28:57.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48f57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.018+0000 7f48ff0ff640 10 monclient(hunting): authenticate will time out at 1467.158596s 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766527048 0 0) 0x7f48f8139af0 con 0x7f48f813f650 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48dc004eb0 con 0x7f48f813f650 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122627225 0 0) 0x7f48f8139c80 con 0x7f48f80a9d70 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f8139af0 con 0x7f48f80a9d70 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813875763 0 0) 0x7f48dc004eb0 con 0x7f48f813f650 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f8139c80 con 0x7f48f813f650 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f48e4001510 con 0x7f48f813f650 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): dump: 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:57.019 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.021+0000 7f48f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3025380700 0 0) 0x7f48f8139af0 con 0x7f48f80a9d70 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48dc004eb0 con 0x7f48f80a9d70 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f48ec001c30 con 0x7f48f80a9d70 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient(hunting): dump: 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.020 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3285633990 0 0) 0x7f48f8139c80 con 0x7f48f813f650 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 >> v1:172.21.15.52:6790/0 conn(0x7f48f80a9d70 legacy=0x7f48f813b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 >> v1:172.21.15.52:6789/0 conn(0x7f48f81457f0 legacy=0x7f48f80b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 monclient: found mon.b 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8138690 con 0x7f48f813f650 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 10 monclient: _renew_subs 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f8138940 con 0x7f48f813f650 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 10 monclient: _renew_subs 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f81d7350 con 0x7f48f813f650 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.022+0000 7f48ff0ff640 1 librados: init done 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.023+0000 7f48ff0ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.023+0000 7f48ff0ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.023+0000 7f48ff0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.023+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f8075f10 con 0x7f48f813f650 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48e4001a20 con 0x7f48f813f650 2024-03-29T11:28:57.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f48e4003910 con 0x7f48f813f650 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f57fa640 10 monclient: dump: 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:28:57.023 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.024+0000 7f48f6ffd640 4 set_mon_vals no callback set 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.025+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f48e40198e0 con 0x7f48f813f650 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.026+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f48e4083750 con 0x7f48f813f650 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.026+0000 7f48f57fa640 10 client.4510.objecter ms_dispatch 0x7f48f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.026+0000 7f48f57fa640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.026+0000 7f48f57fa640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.026+0000 7f48f57fa640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.026+0000 7f48fce9b640 10 monclient: get_auth_request con 0x7f48dc06ddf0 auth_method 0 2024-03-29T11:28:57.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.027+0000 7f48f57fa640 10 client.4510.objecter ms_handle_connect 0x7f48dc06ddf0 2024-03-29T11:28:57.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.033+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48e404f030 con 0x7f48f813f650 2024-03-29T11:28:57.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.033+0000 7f48f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.033+0000 7f48f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:57.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.344+0000 7f48ff0ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.344+0000 7f48ff0ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.344+0000 7f48ff0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:28:57.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.344+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48f80b3680 con 0x7f48f813f650 2024-03-29T11:28:57.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.345+0000 7f48f57fa640 1 -- 172.21.15.52:0/321409978 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f48e4050f60 con 0x7f48f813f650 2024-03-29T11:28:57.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.345+0000 7f48f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.345+0000 7f48f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:57.336 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:28:57.336 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 10 librados: watch_flush enter 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 10 librados: watch_flush exit 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 >> v1:172.21.15.79:6816/407480584 conn(0x7f48dc06ddf0 legacy=0x7f48dc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 10 monclient: shutdown 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 >> v1:172.21.15.79:6789/0 conn(0x7f48f813f650 legacy=0x7f48f81393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 shutdown_connections 2024-03-29T11:28:57.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.349+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 >> 172.21.15.52:0/321409978 conn(0x7f48f8066a00 msgr2=0x7f48f80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.350+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 shutdown_connections 2024-03-29T11:28:57.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.350+0000 7f48ff0ff640 1 -- 172.21.15.52:0/321409978 wait complete. 2024-03-29T11:28:57.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:28:57.350+0000 7f48ff0ff640 1 librados: shutdown 2024-03-29T11:28:57.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:04.162 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:04.317 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.326+0000 7f34213ba640 10 monclient: get_monmap_and_config 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.326+0000 7f34213ba640 10 monclient: build_initial_monmap 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.326+0000 7f34213ba640 10 monclient: monmap: 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:04.327659+0000 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:04.327659+0000 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:04.318 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.328+0000 7f34213ba640 1 Processor -- start 2024-03-29T11:29:04.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.328+0000 7f34213ba640 1 -- start start 2024-03-29T11:29:04.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.328+0000 7f34213ba640 10 monclient: init 2024-03-29T11:29:04.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.329+0000 7f34213ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:04.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.329+0000 7f34213ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:04.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.329+0000 7f34213ba640 10 monclient(hunting): picked mon.noname-a con 0x7f341c0a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:04.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 10 monclient(hunting): picked mon.noname-b con 0x7f341c0af2e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:04.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 10 monclient(hunting): picked mon.noname-c con 0x7f341c141ef0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c0a9b30 con 0x7f341c0a9750 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c0ac4f0 con 0x7f341c141ef0 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c0af770 con 0x7f341c0af2e0 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.331+0000 7f34213ba640 10 monclient(hunting): authenticate will time out at 1474.471762s 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.332+0000 7f3413fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f341c0af2e0 0x7f341c1417e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:47194/0 (socket says 172.21.15.52:47194) 2024-03-29T11:29:04.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.332+0000 7f3413fff640 1 -- 172.21.15.52:0/3889780336 learned_addr learned my addr 172.21.15.52:0/3889780336 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703800504 0 0) 0x7f341c0a9b30 con 0x7f341c0a9750 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3408005730 con 0x7f341c0a9750 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039744113 0 0) 0x7f341c0af770 con 0x7f341c0af2e0 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:29:04.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.333+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3408006ab0 con 0x7f341c0af2e0 2024-03-29T11:29:04.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.334+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632221811 0 0) 0x7f3408005730 con 0x7f341c0a9750 2024-03-29T11:29:04.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.334+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 >> v1:172.21.15.52:6790/0 conn(0x7f341c0af2e0 legacy=0x7f341c1417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.334+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 >> v1:172.21.15.79:6789/0 conn(0x7f341c141ef0 legacy=0x7f341c1498e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c0b00c0 con 0x7f341c0a9750 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:04.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3404001670 con 0x7f341c0a9750 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.335+0000 7f341a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 10 monclient: dump: 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.329 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f34213ba640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f34213ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3404001850 con 0x7f341c0a9750 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 1 -- 172.21.15.52:0/3889780336 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3404001be0 con 0x7f341c0a9750 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f341a7fc640 10 monclient: dump: 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:04.330 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:04.331 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:04.331 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f34213ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:04.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.336+0000 7f34213ba640 4 set_mon_vals no callback set 2024-03-29T11:29:04.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.338+0000 7f34213ba640 10 monclient: shutdown 2024-03-29T11:29:04.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.340+0000 7f34213ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:04.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.340+0000 7f34213ba640 1 -- 172.21.15.52:0/3889780336 >> v1:172.21.15.52:6789/0 conn(0x7f341c0a9750 legacy=0x7f341c13b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.340+0000 7f34213ba640 1 -- 172.21.15.52:0/3889780336 shutdown_connections 2024-03-29T11:29:04.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.340+0000 7f34213ba640 1 -- 172.21.15.52:0/3889780336 >> 172.21.15.52:0/3889780336 conn(0x7f341c0b3a00 msgr2=0x7f341c0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:04.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.341+0000 7f34213ba640 1 -- 172.21.15.52:0/3889780336 shutdown_connections 2024-03-29T11:29:04.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.344+0000 7f34213ba640 1 -- 172.21.15.52:0/3889780336 wait complete. 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.344+0000 7f34213ba640 10 monclient: build_initial_monmap 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.344+0000 7f34213ba640 10 monclient: monmap: 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:04.335 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:04.336 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:04.336 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 1 librados: starting msgr at 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 1 librados: starting objecter 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 1 Processor -- start 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 1 -- start start 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 1 librados: setting wanted keys 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 1 librados: calling monclient init 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.345+0000 7f34213ba640 10 monclient: init 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.347+0000 7f34213ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:04.337 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.347+0000 7f34213ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f34213ba640 10 monclient(hunting): picked mon.noname-b con 0x7f341c0a9750 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f341affd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f341c0a9750 0x7f341c1d2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:58491/0 (socket says 172.21.15.52:58491) 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f341affd640 1 -- 172.21.15.52:0/2920804649 learned_addr learned my addr 172.21.15.52:0/2920804649 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f34213ba640 10 monclient(hunting): picked mon.noname-a con 0x7f341c141ef0 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f34213ba640 10 monclient(hunting): picked mon.noname-c con 0x7f341c1d3170 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c06e0b0 con 0x7f341c141ef0 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c13e120 con 0x7f341c0a9750 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.349+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c13e640 con 0x7f341c1d3170 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34213ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34213ba640 10 monclient(hunting): authenticate will time out at 1474.490126s 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 client.?.objecter ms_handle_connect 0x7f341c0a9750 2024-03-29T11:29:04.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 client.?.objecter ms_handle_connect 0x7f341c1d3170 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214040886 0 0) 0x7f341c13e640 con 0x7f341c1d3170 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc004eb0 con 0x7f341c1d3170 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185327735 0 0) 0x7f341c13e120 con 0x7f341c0a9750 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.350+0000 7f34137fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:29:04.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.351+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c13e640 con 0x7f341c0a9750 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.353+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1921330371 0 0) 0x7f341c13e640 con 0x7f341c0a9750 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.353+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f341c13e120 con 0x7f341c0a9750 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3404001be0 con 0x7f341c0a9750 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): dump: 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805641060 0 0) 0x7f33fc004eb0 con 0x7f341c1d3170 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f341c13e640 con 0x7f341c1d3170 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3414001aa0 con 0x7f341c1d3170 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient(hunting): dump: 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2312287803 0 0) 0x7f341c13e120 con 0x7f341c0a9750 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 >> v1:172.21.15.52:6790/0 conn(0x7f341c1d3170 legacy=0x7f341c1de380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 >> v1:172.21.15.52:6789/0 conn(0x7f341c141ef0 legacy=0x7f341c1338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 1 monclient: found mon.b 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.354+0000 7f34137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.355+0000 7f34137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.355+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c13e7d0 con 0x7f341c0a9750 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.355+0000 7f34137fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.355+0000 7f34137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.355+0000 7f34137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:04.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:29:04.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 10 monclient: _renew_subs 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c13d2a0 con 0x7f341c0a9750 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 10 monclient: _renew_subs 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c13dd20 con 0x7f341c0a9750 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.356+0000 7f34213ba640 1 librados: init done 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.357+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3404001d40 con 0x7f341c0a9750 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.357+0000 7f34137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3404003a40 con 0x7f341c0a9750 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34137fe640 10 monclient: dump: 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34213ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34213ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34213ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c13d490 con 0x7f341c0a9750 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f3419ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.358+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f3404003c60 con 0x7f341c0a9750 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.359+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f3404084a40 con 0x7f341c0a9750 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.359+0000 7f34137fe640 10 client.4513.objecter ms_dispatch 0x7f341c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.359+0000 7f34137fe640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.359+0000 7f34137fe640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.359+0000 7f34137fe640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.360+0000 7f3413fff640 10 monclient: get_auth_request con 0x7f33fc06ddb0 auth_method 0 2024-03-29T11:29:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.360+0000 7f34137fe640 10 client.4513.objecter ms_handle_connect 0x7f33fc06ddb0 2024-03-29T11:29:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.367+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3404050320 con 0x7f341c0a9750 2024-03-29T11:29:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.367+0000 7f34137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.367+0000 7f34137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:04.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.694+0000 7f34213ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.694+0000 7f34213ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.694+0000 7f34213ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:04.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.694+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f341c06def0 con 0x7f341c0a9750 2024-03-29T11:29:04.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.696+0000 7f34137fe640 1 -- 172.21.15.52:0/2920804649 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f3404052250 con 0x7f341c0a9750 2024-03-29T11:29:04.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.696+0000 7f34137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.696+0000 7f34137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:04.687 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:04.687 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.700+0000 7f34213ba640 10 librados: watch_flush enter 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 10 librados: watch_flush exit 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 >> v1:172.21.15.79:6816/407480584 conn(0x7f33fc06ddb0 legacy=0x7f33fc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 10 monclient: shutdown 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.702+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 >> v1:172.21.15.79:6789/0 conn(0x7f341c0a9750 legacy=0x7f341c1d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.704+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 shutdown_connections 2024-03-29T11:29:04.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.704+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 >> 172.21.15.52:0/2920804649 conn(0x7f341c0b3a00 msgr2=0x7f341c0b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:04.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.705+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 shutdown_connections 2024-03-29T11:29:04.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.705+0000 7f34213ba640 1 -- 172.21.15.52:0/2920804649 wait complete. 2024-03-29T11:29:04.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:04.708+0000 7f34213ba640 1 librados: shutdown 2024-03-29T11:29:04.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:11.524 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:11.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.679+0000 7f2d6a95d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:11.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.679+0000 7f2d6a95d640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.679+0000 7f2d6a95d640 10 monclient: monmap: 2024-03-29T11:29:11.671 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:11.672 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:11.672 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:11.680078+0000 2024-03-29T11:29:11.672 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:11.680078+0000 2024-03-29T11:29:11.672 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.672 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:11.672 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:11.673 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:11.673 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:11.673 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:11.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.682+0000 7f2d6a95d640 1 Processor -- start 2024-03-29T11:29:11.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.682+0000 7f2d6a95d640 1 -- start start 2024-03-29T11:29:11.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.682+0000 7f2d6a95d640 10 monclient: init 2024-03-29T11:29:11.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d640a9ac0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6413ee90 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d640af900 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d640afd10 con 0x7f2d6413ee90 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d640af6f0 con 0x7f2d640a9ac0 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d640aac80 con 0x7f2d640af900 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.683+0000 7f2d6a95d640 10 monclient(hunting): authenticate will time out at 1481.823947s 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.684+0000 7f2d68efa640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f2d640af900 0x7f2d64149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:54688/0 (socket says 172.21.15.52:54688) 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.684+0000 7f2d68efa640 1 -- 172.21.15.52:0/655431947 learned_addr learned my addr 172.21.15.52:0/655431947 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.686+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437766255 0 0) 0x7f2d640aac80 con 0x7f2d640af900 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.686+0000 7f2d62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.686+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.686+0000 7f2d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.686+0000 7f2d62ffd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.686+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d4c003520 con 0x7f2d640af900 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2901038902 0 0) 0x7f2d4c003520 con 0x7f2d640af900 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 >> v1:172.21.15.79:6789/0 conn(0x7f2d640a9ac0 legacy=0x7f2d640adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 >> v1:172.21.15.52:6789/0 conn(0x7f2d6413ee90 legacy=0x7f2d6413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 monclient: found mon.noname-b 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d640aaf60 con 0x7f2d640af900 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2d5c001a10 con 0x7f2d640af900 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d6a95d640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d6a95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: dump: 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d6a95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d5c001ca0 con 0x7f2d640af900 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d6a95d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d6a95d640 4 set_mon_vals no callback set 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.688+0000 7f2d62ffd640 1 -- 172.21.15.52:0/655431947 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2d5c002030 con 0x7f2d640af900 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.689+0000 7f2d62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.689+0000 7f2d62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.689+0000 7f2d62ffd640 10 monclient: dump: 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.689+0000 7f2d6a95d640 10 monclient: shutdown 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.689+0000 7f2d6a95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.689+0000 7f2d6a95d640 1 -- 172.21.15.52:0/655431947 >> v1:172.21.15.52:6790/0 conn(0x7f2d640af900 legacy=0x7f2d64149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.691+0000 7f2d6a95d640 1 -- 172.21.15.52:0/655431947 shutdown_connections 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.691+0000 7f2d6a95d640 1 -- 172.21.15.52:0/655431947 >> 172.21.15.52:0/655431947 conn(0x7f2d640b35e0 msgr2=0x7f2d640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.693+0000 7f2d6a95d640 1 -- 172.21.15.52:0/655431947 shutdown_connections 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.694+0000 7f2d6a95d640 1 -- 172.21.15.52:0/655431947 wait complete. 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.694+0000 7f2d6a95d640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.694+0000 7f2d6a95d640 10 monclient: monmap: 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.686 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 1 librados: starting msgr at 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 1 librados: starting objecter 2024-03-29T11:29:11.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 1 Processor -- start 2024-03-29T11:29:11.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 1 -- start start 2024-03-29T11:29:11.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 1 librados: setting wanted keys 2024-03-29T11:29:11.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 1 librados: calling monclient init 2024-03-29T11:29:11.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.695+0000 7f2d6a95d640 10 monclient: init 2024-03-29T11:29:11.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:11.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d640a9ac0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:11.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d640af900 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:11.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6413ee90 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d64141af0 con 0x7f2d6413ee90 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d64071380 con 0x7f2d640a9ac0 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d64071ab0 con 0x7f2d640af900 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.696+0000 7f2d6a95d640 10 monclient(hunting): authenticate will time out at 1481.836754s 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.697+0000 7f2d637fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f2d640af900 0x7f2d6406ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56246/0 (socket says 172.21.15.52:56246) 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.697+0000 7f2d637fe640 1 -- 172.21.15.52:0/2053468990 learned_addr learned my addr 172.21.15.52:0/2053468990 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:11.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.699+0000 7f2d60ff9640 10 client.?.objecter ms_handle_connect 0x7f2d640af900 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.699+0000 7f2d60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.700+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225257610 0 0) 0x7f2d64071ab0 con 0x7f2d640af900 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.700+0000 7f2d60ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.700+0000 7f2d60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.700+0000 7f2d60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.700+0000 7f2d60ff9640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.700+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d50004eb0 con 0x7f2d640af900 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338532611 0 0) 0x7f2d50004eb0 con 0x7f2d640af900 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d64071ab0 con 0x7f2d640af900 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2d540017d0 con 0x7f2d640af900 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2734137056 0 0) 0x7f2d64071ab0 con 0x7f2d640af900 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 >> v1:172.21.15.79:6789/0 conn(0x7f2d640a9ac0 legacy=0x7f2d64071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 >> v1:172.21.15.52:6789/0 conn(0x7f2d6413ee90 legacy=0x7f2d641dd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 monclient: found mon.c 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.701+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d640717a0 con 0x7f2d640af900 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 10 monclient: _renew_subs 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6406f430 con 0x7f2d640af900 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 10 monclient: _renew_subs 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6406ff50 con 0x7f2d640af900 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d6a95d640 1 librados: init done 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d54001930 con 0x7f2d640af900 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2d54004a00 con 0x7f2d640af900 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d60ff9640 10 monclient: dump: 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.702+0000 7f2d627fc640 4 set_mon_vals no callback set 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f2d5401a9d0 con 0x7f2d640af900 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d6a95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d6a95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d6a95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d44005080 con 0x7f2d640af900 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f2d54084a50 con 0x7f2d640af900 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d60ff9640 10 client.4604.objecter ms_dispatch 0x7f2d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d60ff9640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:11.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.708+0000 7f2d60ff9640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:11.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.709+0000 7f2d60ff9640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:29:11.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.709+0000 7f2d63fff640 10 monclient: get_auth_request con 0x7f2d5006dc70 auth_method 0 2024-03-29T11:29:11.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.709+0000 7f2d60ff9640 10 client.4604.objecter ms_handle_connect 0x7f2d5006dc70 2024-03-29T11:29:11.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.716+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d54050330 con 0x7f2d640af900 2024-03-29T11:29:11.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.716+0000 7f2d60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:11.716+0000 7f2d60ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:12.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.047+0000 7f2d6a95d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.047+0000 7f2d6a95d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.047+0000 7f2d6a95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:12.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.047+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d44005330 con 0x7f2d640af900 2024-03-29T11:29:12.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.048+0000 7f2d60ff9640 1 -- 172.21.15.52:0/2053468990 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f2d54052260 con 0x7f2d640af900 2024-03-29T11:29:12.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.048+0000 7f2d60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.048+0000 7f2d60ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:12.043 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:12.043 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:12.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 10 librados: watch_flush enter 2024-03-29T11:29:12.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 10 librados: watch_flush exit 2024-03-29T11:29:12.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:29:12.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:29:12.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 >> v1:172.21.15.79:6816/407480584 conn(0x7f2d5006dc70 legacy=0x7f2d50070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 10 monclient: shutdown 2024-03-29T11:29:12.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:12.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.052+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 >> v1:172.21.15.52:6790/0 conn(0x7f2d640af900 legacy=0x7f2d6406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.056+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 shutdown_connections 2024-03-29T11:29:12.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.056+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 >> 172.21.15.52:0/2053468990 conn(0x7f2d640b35e0 msgr2=0x7f2d6412aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:12.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.056+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 shutdown_connections 2024-03-29T11:29:12.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.056+0000 7f2d6a95d640 1 -- 172.21.15.52:0/2053468990 wait complete. 2024-03-29T11:29:12.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:12.057+0000 7f2d6a95d640 1 librados: shutdown 2024-03-29T11:29:12.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:18.873 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.034+0000 7f6185233640 10 monclient: get_monmap_and_config 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.034+0000 7f6185233640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.034+0000 7f6185233640 10 monclient: monmap: 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:19.035614+0000 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:19.035614+0000 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:19.026 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:19.027 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.035+0000 7f6185233640 1 Processor -- start 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.035+0000 7f6185233640 1 -- start start 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.035+0000 7f6185233640 10 monclient: init 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient(hunting): picked mon.noname-b con 0x7f61801422b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient(hunting): picked mon.noname-a con 0x7f6180142690 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient(hunting): picked mon.noname-c con 0x7f61800b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180142c80 con 0x7f6180142690 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61800aae40 con 0x7f61800b17d0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618013fbd0 con 0x7f61801422b0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.036+0000 7f6185233640 10 monclient(hunting): authenticate will time out at 1489.176763s 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617f59e640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f61800b17d0 0x7f61800b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:52950/0 (socket says 172.21.15.52:52950) 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617f59e640 1 -- 172.21.15.52:0/784836761 learned_addr learned my addr 172.21.15.52:0/784836761 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809564883 0 0) 0x7f61800aae40 con 0x7f61800b17d0 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617dd9b640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.037+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6168003520 con 0x7f61800b17d0 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884168325 0 0) 0x7f6180142c80 con 0x7f6180142690 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61800aae40 con 0x7f6180142690 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.038+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494150813 0 0) 0x7f6168003520 con 0x7f61800b17d0 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 >> v1:172.21.15.52:6790/0 conn(0x7f61801422b0 legacy=0x7f618013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 >> v1:172.21.15.52:6789/0 conn(0x7f6180142690 legacy=0x7f61800b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180151940 con 0x7f61800b17d0 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6178001a10 con 0x7f61800b17d0 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: dump: 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:19.036 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f6185233640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f6185233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6178001c30 con 0x7f61800b17d0 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 1 -- 172.21.15.52:0/784836761 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6178001f40 con 0x7f61800b17d0 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f617dd9b640 10 monclient: dump: 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f6185233640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.039+0000 7f6185233640 4 set_mon_vals no callback set 2024-03-29T11:29:19.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 10 monclient: shutdown 2024-03-29T11:29:19.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 1 -- 172.21.15.52:0/784836761 >> v1:172.21.15.79:6789/0 conn(0x7f61800b17d0 legacy=0x7f61800b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 1 -- 172.21.15.52:0/784836761 shutdown_connections 2024-03-29T11:29:19.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 1 -- 172.21.15.52:0/784836761 >> 172.21.15.52:0/784836761 conn(0x7f6180066a00 msgr2=0x7f6180066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 1 -- 172.21.15.52:0/784836761 shutdown_connections 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.040+0000 7f6185233640 1 -- 172.21.15.52:0/784836761 wait complete. 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.041+0000 7f6185233640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.041+0000 7f6185233640 10 monclient: monmap: 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:19.040 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:19.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 1 librados: starting msgr at 2024-03-29T11:29:19.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 1 librados: starting objecter 2024-03-29T11:29:19.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 1 Processor -- start 2024-03-29T11:29:19.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 1 -- start start 2024-03-29T11:29:19.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 1 librados: setting wanted keys 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 1 librados: calling monclient init 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.042+0000 7f6185233640 10 monclient: init 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.043+0000 7f6185233640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.043+0000 7f6185233640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.043+0000 7f6185233640 10 monclient(hunting): picked mon.noname-a con 0x7f61800b17d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 10 monclient(hunting): picked mon.noname-b con 0x7f61801422b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 10 monclient(hunting): picked mon.noname-c con 0x7f6180142690 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61801de8d0 con 0x7f61800b17d0 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61801debb0 con 0x7f61801422b0 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f617e59c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f61801422b0 0x7f6180070380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:54438/0 (socket says 172.21.15.52:54438) 2024-03-29T11:29:19.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f617e59c640 1 -- 172.21.15.52:0/1906717832 learned_addr learned my addr 172.21.15.52:0/1906717832 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61801df750 con 0x7f6180142690 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 client.?.objecter ms_handle_connect 0x7f61801422b0 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f6185233640 10 monclient(hunting): authenticate will time out at 1489.184572s 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 client.?.objecter ms_handle_connect 0x7f61800b17d0 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 client.?.objecter ms_handle_connect 0x7f6180142690 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925928290 0 0) 0x7f61801debb0 con 0x7f61801422b0 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c004eb0 con 0x7f61801422b0 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877238072 0 0) 0x7f61801de8d0 con 0x7f61800b17d0 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.044+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61801debb0 con 0x7f61800b17d0 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 443210636 0 0) 0x7f616c004eb0 con 0x7f61801422b0 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61801de8d0 con 0x7f61801422b0 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f6174001970 con 0x7f61801422b0 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): dump: 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:19.048 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109285486 0 0) 0x7f61801df750 con 0x7f6180142690 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c004eb0 con 0x7f6180142690 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1921150493 0 0) 0x7f61801de8d0 con 0x7f61801422b0 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 >> v1:172.21.15.52:6790/0 conn(0x7f6180142690 legacy=0x7f61801e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 >> v1:172.21.15.52:6789/0 conn(0x7f61800b17d0 legacy=0x7f61801df040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 monclient: found mon.b 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618006ec20 con 0x7f61801422b0 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f61677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f6185233640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f6185233640 10 monclient: _renew_subs 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.045+0000 7f6185233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:19.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61801316a0 con 0x7f61801422b0 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f6185233640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f6185233640 10 monclient: _renew_subs 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f6185233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61800b5420 con 0x7f61801422b0 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f6185233640 1 librados: init done 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61740035f0 con 0x7f61801422b0 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f61677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f61740039c0 con 0x7f61801422b0 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f61677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f61677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f61677fe640 10 monclient: dump: 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:19.052 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.046+0000 7f617cd99640 4 set_mon_vals no callback set 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.047+0000 7f6185233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.047+0000 7f6185233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.047+0000 7f6185233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.047+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6160005080 con 0x7f61801422b0 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.048+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f6174003ba0 con 0x7f61801422b0 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f6174084ad0 con 0x7f61801422b0 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f61677fe640 10 client.4519.objecter ms_dispatch 0x7f6180000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f61677fe640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f61677fe640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f617ed9d640 10 monclient: get_auth_request con 0x7f616c06de00 auth_method 0 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f61677fe640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.049+0000 7f61677fe640 10 client.4519.objecter ms_handle_connect 0x7f616c06de00 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.055+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61740503b0 con 0x7f61801422b0 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.055+0000 7f61677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.055+0000 7f61677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:19.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.465+0000 7f6185233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.465+0000 7f6185233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.465+0000 7f6185233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:29:19.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.465+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6160005330 con 0x7f61801422b0 2024-03-29T11:29:19.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.466+0000 7f61677fe640 1 -- 172.21.15.52:0/1906717832 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f61740522e0 con 0x7f61801422b0 2024-03-29T11:29:19.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.466+0000 7f61677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.466+0000 7f61677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:19.459 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:19.459 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:19.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 10 librados: watch_flush enter 2024-03-29T11:29:19.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 10 librados: watch_flush exit 2024-03-29T11:29:19.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 10 client.4519.objecter successfully canceled tick 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 >> v1:172.21.15.79:6816/407480584 conn(0x7f616c06de00 legacy=0x7f616c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 10 monclient: shutdown 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 >> v1:172.21.15.79:6789/0 conn(0x7f61801422b0 legacy=0x7f6180070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 shutdown_connections 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 >> 172.21.15.52:0/1906717832 conn(0x7f6180066a00 msgr2=0x7f61800b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 shutdown_connections 2024-03-29T11:29:19.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 -- 172.21.15.52:0/1906717832 wait complete. 2024-03-29T11:29:19.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:19.470+0000 7f6185233640 1 librados: shutdown 2024-03-29T11:29:19.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:26.285 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.445+0000 7f676a588640 10 monclient: get_monmap_and_config 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.445+0000 7f676a588640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.446+0000 7f676a588640 10 monclient: monmap: 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:26.446939+0000 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:26.446939+0000 2024-03-29T11:29:26.437 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.438 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:26.438 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:26.438 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:26.438 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:26.438 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:26.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.448+0000 7f676a588640 1 Processor -- start 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.448+0000 7f676a588640 1 -- start start 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.448+0000 7f676a588640 10 monclient: init 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient(hunting): picked mon.noname-a con 0x7f67640b4330 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient(hunting): picked mon.noname-b con 0x7f67641470b0 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:26.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient(hunting): picked mon.noname-c con 0x7f67641439b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676406f2a0 con 0x7f67640b4330 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6764146b90 con 0x7f67641439b0 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640b2ca0 con 0x7f67641470b0 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.449+0000 7f676a588640 10 monclient(hunting): authenticate will time out at 1496.589548s 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.456+0000 7f67637fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f67641470b0 0x7f67640b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41889/0 (socket says 172.21.15.52:41889) 2024-03-29T11:29:26.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.456+0000 7f67637fe640 1 -- 172.21.15.52:0/223637243 learned_addr learned my addr 172.21.15.52:0/223637243 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716136088 0 0) 0x7f67640b2ca0 con 0x7f67641470b0 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f674c003520 con 0x7f67641470b0 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326750402 0 0) 0x7f6764146b90 con 0x7f67641439b0 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.460+0000 7f6762ffd640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:29:26.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67640b2ca0 con 0x7f67641439b0 2024-03-29T11:29:26.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533980337 0 0) 0x7f676406f2a0 con 0x7f67640b4330 2024-03-29T11:29:26.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:29:26.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6764146b90 con 0x7f67640b4330 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892543957 0 0) 0x7f67640b2ca0 con 0x7f67641439b0 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 >> v1:172.21.15.52:6790/0 conn(0x7f67641470b0 legacy=0x7f67640b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 >> v1:172.21.15.52:6789/0 conn(0x7f67640b4330 legacy=0x7f67640b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 monclient: found mon.noname-c 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676406e0c0 con 0x7f67641439b0 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.461+0000 7f6762ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f67540015a0 con 0x7f67641439b0 2024-03-29T11:29:26.454 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f6762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f6762ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f6762ffd640 10 monclient: dump: 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f676a588640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:29:26.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f676a588640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f6762ffd640 1 -- 172.21.15.52:0/223637243 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67540017c0 con 0x7f67641439b0 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f6762ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f676a588640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.462+0000 7f676a588640 4 set_mon_vals no callback set 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.463+0000 7f676a588640 10 monclient: shutdown 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.463+0000 7f676a588640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.463+0000 7f676a588640 1 -- 172.21.15.52:0/223637243 >> v1:172.21.15.79:6789/0 conn(0x7f67641439b0 legacy=0x7f6764151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.463+0000 7f676a588640 1 -- 172.21.15.52:0/223637243 shutdown_connections 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.463+0000 7f676a588640 1 -- 172.21.15.52:0/223637243 >> 172.21.15.52:0/223637243 conn(0x7f6764066a00 msgr2=0x7f6764066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.463+0000 7f676a588640 1 -- 172.21.15.52:0/223637243 shutdown_connections 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.464+0000 7f676a588640 1 -- 172.21.15.52:0/223637243 wait complete. 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.464+0000 7f676a588640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 10 monclient: monmap: 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:26.458 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:26.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 1 librados: starting msgr at 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 1 librados: starting objecter 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 1 Processor -- start 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 1 -- start start 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 1 librados: setting wanted keys 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 1 librados: calling monclient init 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.465+0000 7f676a588640 10 monclient: init 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.466+0000 7f676a588640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.466+0000 7f676a588640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:26.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.466+0000 7f676a588640 10 monclient(hunting): picked mon.noname-c con 0x7f67640b4330 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 10 monclient(hunting): picked mon.noname-a con 0x7f67641439b0 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 10 monclient(hunting): picked mon.noname-b con 0x7f67641470b0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640ac3f0 con 0x7f67641439b0 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640ad570 con 0x7f67641470b0 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640abed0 con 0x7f67640b4330 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.470+0000 7f676a588640 10 monclient(hunting): authenticate will time out at 1496.610813s 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f67637fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f67641439b0 0x7f67640b0d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:57799/0 (socket says 172.21.15.52:57799) 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f67637fe640 1 -- 172.21.15.52:0/539825948 learned_addr learned my addr 172.21.15.52:0/539825948 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 10 client.?.objecter ms_handle_connect 0x7f67641439b0 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559448539 0 0) 0x7f67640ac3f0 con 0x7f67641439b0 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:29:26.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.472+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6750004eb0 con 0x7f67641439b0 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2444640221 0 0) 0x7f6750004eb0 con 0x7f67641439b0 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67640ac3f0 con 0x7f67641439b0 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f675c001c90 con 0x7f67641439b0 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:26.466 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2131241081 0 0) 0x7f67640ac3f0 con 0x7f67641439b0 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 >> v1:172.21.15.52:6790/0 conn(0x7f67640b4330 legacy=0x7f67641e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.474+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 >> v1:172.21.15.79:6789/0 conn(0x7f67641470b0 legacy=0x7f67641ef5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:26.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 1 monclient: found mon.a 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640aba40 con 0x7f67641439b0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f6760ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f676a588640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f676a588640 10 monclient: _renew_subs 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f676a588640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67640abc90 con 0x7f67641439b0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.476+0000 7f676a588640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f676a588640 10 monclient: _renew_subs 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f676a588640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6764146000 con 0x7f67641439b0 2024-03-29T11:29:26.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f676a588640 1 librados: init done 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f675c0020e0 con 0x7f67641439b0 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f6760ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f675c004fb0 con 0x7f67641439b0 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f6760ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f6760ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.477+0000 7f6760ff9640 10 monclient: dump: 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:26.471 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:26.472 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.478+0000 7f67627fc640 4 set_mon_vals no callback set 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.479+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f675c005110 con 0x7f67641439b0 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.480+0000 7f676a588640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.480+0000 7f676a588640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.480+0000 7f676a588640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.480+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6744005080 con 0x7f67641439b0 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.480+0000 7f6763fff640 10 monclient: get_auth_request con 0x7f675006dcb0 auth_method 0 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.481+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f675c084de0 con 0x7f67641439b0 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.481+0000 7f6760ff9640 10 client.4542.objecter ms_dispatch 0x7f6764000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.481+0000 7f6760ff9640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.481+0000 7f6760ff9640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:26.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.481+0000 7f6760ff9640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:29:26.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.481+0000 7f6760ff9640 10 client.4542.objecter ms_handle_connect 0x7f675006dcb0 2024-03-29T11:29:26.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.493+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f675c0506c0 con 0x7f67641439b0 2024-03-29T11:29:26.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.493+0000 7f6760ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.493+0000 7f6760ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:26.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.827+0000 7f676a588640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.827+0000 7f676a588640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.827+0000 7f676a588640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:26.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.827+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6744005330 con 0x7f67641439b0 2024-03-29T11:29:26.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.829+0000 7f6760ff9640 1 -- 172.21.15.52:0/539825948 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f675c0525f0 con 0x7f67641439b0 2024-03-29T11:29:26.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.829+0000 7f6760ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.829+0000 7f6760ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:26.821 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:26.821 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:26.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 10 librados: watch_flush enter 2024-03-29T11:29:26.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 10 librados: watch_flush exit 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 >> v1:172.21.15.79:6816/407480584 conn(0x7f675006dcb0 legacy=0x7f6750070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 10 monclient: shutdown 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 >> v1:172.21.15.52:6789/0 conn(0x7f67641439b0 legacy=0x7f67640b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 shutdown_connections 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.833+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 >> 172.21.15.52:0/539825948 conn(0x7f6764066a00 msgr2=0x7f67641d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.834+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 shutdown_connections 2024-03-29T11:29:26.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.835+0000 7f676a588640 1 -- 172.21.15.52:0/539825948 wait complete. 2024-03-29T11:29:26.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:26.836+0000 7f676a588640 1 librados: shutdown 2024-03-29T11:29:26.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:33.652 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.811+0000 7f3560e64640 10 monclient: get_monmap_and_config 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.811+0000 7f3560e64640 10 monclient: build_initial_monmap 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.811+0000 7f3560e64640 10 monclient: monmap: 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:33.812380+0000 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:33.812380+0000 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:33.803 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.812+0000 7f3560e64640 1 Processor -- start 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.812+0000 7f3560e64640 1 -- start start 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.812+0000 7f3560e64640 10 monclient: init 2024-03-29T11:29:33.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient(hunting): picked mon.noname-c con 0x7f355c06d9d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient(hunting): picked mon.noname-b con 0x7f355c13e820 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient(hunting): picked mon.noname-a con 0x7f355c0af1d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0aed10 con 0x7f355c0af1d0 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0afd50 con 0x7f355c06d9d0 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0ab490 con 0x7f355c13e820 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.813+0000 7f3560e64640 10 monclient(hunting): authenticate will time out at 1503.953825s 2024-03-29T11:29:33.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.814+0000 7f355ad9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f355c0af1d0 0x7f355c0af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:49966/0 (socket says 172.21.15.52:49966) 2024-03-29T11:29:33.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.814+0000 7f355ad9d640 1 -- 172.21.15.52:0/4145706231 learned_addr learned my addr 172.21.15.52:0/4145706231 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:33.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.815+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746065903 0 0) 0x7f355c0aed10 con 0x7f355c0af1d0 2024-03-29T11:29:33.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.815+0000 7f355959a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.815+0000 7f355959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.815+0000 7f355959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.815+0000 7f355959a640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:29:33.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.815+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3544003520 con 0x7f355c0af1d0 2024-03-29T11:29:33.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.816+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210571221 0 0) 0x7f3544003520 con 0x7f355c0af1d0 2024-03-29T11:29:33.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.816+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 >> v1:172.21.15.52:6790/0 conn(0x7f355c13e820 legacy=0x7f355c13ec50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 >> v1:172.21.15.79:6789/0 conn(0x7f355c06d9d0 legacy=0x7f355c0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 1 monclient: found mon.noname-a 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0ab770 con 0x7f355c0af1d0 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: _finish_auth 0 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3554001a10 con 0x7f355c0af1d0 2024-03-29T11:29:33.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.819+0000 7f355959a640 10 monclient: dump: 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:33.812 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.820+0000 7f3560e64640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.820+0000 7f3560e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3554001be0 con 0x7f355c0af1d0 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f355959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f355959a640 1 -- 172.21.15.52:0/4145706231 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3554001ff0 con 0x7f355c0af1d0 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f3560e64640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f3560e64640 4 set_mon_vals no callback set 2024-03-29T11:29:33.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f355959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f355959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f355959a640 10 monclient: dump: 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f3560e64640 10 monclient: shutdown 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f3560e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:33.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.821+0000 7f3560e64640 1 -- 172.21.15.52:0/4145706231 >> v1:172.21.15.52:6789/0 conn(0x7f355c0af1d0 legacy=0x7f355c0af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.822+0000 7f3560e64640 1 -- 172.21.15.52:0/4145706231 shutdown_connections 2024-03-29T11:29:33.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.822+0000 7f3560e64640 1 -- 172.21.15.52:0/4145706231 >> 172.21.15.52:0/4145706231 conn(0x7f355c06b660 msgr2=0x7f355c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:33.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.824+0000 7f3560e64640 1 -- 172.21.15.52:0/4145706231 shutdown_connections 2024-03-29T11:29:33.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.825+0000 7f3560e64640 1 -- 172.21.15.52:0/4145706231 wait complete. 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 10 monclient: build_initial_monmap 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 10 monclient: monmap: 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:33.817 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:33.818 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:33.818 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:33.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 1 librados: starting msgr at 2024-03-29T11:29:33.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 1 librados: starting objecter 2024-03-29T11:29:33.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 1 Processor -- start 2024-03-29T11:29:33.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 1 -- start start 2024-03-29T11:29:33.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 1 librados: setting wanted keys 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 1 librados: calling monclient init 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.826+0000 7f3560e64640 10 monclient: init 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient(hunting): picked mon.noname-c con 0x7f355c06d9d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient(hunting): picked mon.noname-b con 0x7f355c0af1d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient(hunting): picked mon.noname-a con 0x7f355c13e820 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c06f3a0 con 0x7f355c13e820 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c06f680 con 0x7f355c0af1d0 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c06e730 con 0x7f355c06d9d0 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:33.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.827+0000 7f3560e64640 10 monclient(hunting): authenticate will time out at 1503.967721s 2024-03-29T11:29:33.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.830+0000 7f355ad9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f355c13e820 0x7f355c12ad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:52128/0 (socket says 172.21.15.52:52128) 2024-03-29T11:29:33.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.830+0000 7f355ad9d640 1 -- 172.21.15.52:0/2044075111 learned_addr learned my addr 172.21.15.52:0/2044075111 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:33.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.830+0000 7f354affd640 10 client.?.objecter ms_handle_connect 0x7f355c13e820 2024-03-29T11:29:33.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.830+0000 7f354affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999308369 0 0) 0x7f355c06f3a0 con 0x7f355c13e820 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c004eb0 con 0x7f355c13e820 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168809197 0 0) 0x7f353c004eb0 con 0x7f355c13e820 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f355c06f3a0 con 0x7f355c13e820 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f3554001ef0 con 0x7f355c13e820 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.831+0000 7f354affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:33.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient(hunting): dump: 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:33.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 938432882 0 0) 0x7f355c06f3a0 con 0x7f355c13e820 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 >> v1:172.21.15.52:6790/0 conn(0x7f355c06d9d0 legacy=0x7f355c06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 >> v1:172.21.15.79:6789/0 conn(0x7f355c0af1d0 legacy=0x7f355c06fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 1 monclient: found mon.a 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c138570 con 0x7f355c13e820 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f354affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 10 monclient: _renew_subs 2024-03-29T11:29:33.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:33.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c070570 con 0x7f355c13e820 2024-03-29T11:29:33.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:33.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 10 monclient: _renew_subs 2024-03-29T11:29:33.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:33.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c136c00 con 0x7f355c13e820 2024-03-29T11:29:33.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.832+0000 7f3560e64640 1 librados: init done 2024-03-29T11:29:33.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.833+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3554004f70 con 0x7f355c13e820 2024-03-29T11:29:33.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.833+0000 7f354affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:33.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.833+0000 7f3558d99640 4 set_mon_vals no callback set 2024-03-29T11:29:33.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.834+0000 7f3560e64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.834+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f35540053c0 con 0x7f355c13e820 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.834+0000 7f354affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.834+0000 7f354affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.834+0000 7f354affd640 10 monclient: dump: 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.835+0000 7f3560e64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.835+0000 7f3560e64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:33.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.835+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3540005080 con 0x7f355c13e820 2024-03-29T11:29:33.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.836+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f35540055a0 con 0x7f355c13e820 2024-03-29T11:29:33.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.837+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f3554085070 con 0x7f355c13e820 2024-03-29T11:29:33.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.837+0000 7f354affd640 10 client.4548.objecter ms_dispatch 0x7f355c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:33.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.837+0000 7f354affd640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:33.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.837+0000 7f354affd640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:33.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.837+0000 7f354affd640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T11:29:33.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.839+0000 7f355a59c640 10 monclient: get_auth_request con 0x7f353c06db80 auth_method 0 2024-03-29T11:29:33.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.840+0000 7f354affd640 10 client.4548.objecter ms_handle_connect 0x7f353c06db80 2024-03-29T11:29:33.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.845+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3554050950 con 0x7f355c13e820 2024-03-29T11:29:33.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.845+0000 7f354affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:33.845+0000 7f354affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:34.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.178+0000 7f3560e64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.178+0000 7f3560e64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.178+0000 7f3560e64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:34.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.178+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3540005330 con 0x7f355c13e820 2024-03-29T11:29:34.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.182+0000 7f354affd640 1 -- 172.21.15.52:0/2044075111 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f3554052880 con 0x7f355c13e820 2024-03-29T11:29:34.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.182+0000 7f354affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.182+0000 7f354affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:34.174 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:34.174 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:34.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 10 librados: watch_flush enter 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 10 librados: watch_flush exit 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 10 client.4548.objecter successfully canceled tick 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 >> v1:172.21.15.79:6816/407480584 conn(0x7f353c06db80 legacy=0x7f353c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 10 monclient: shutdown 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.186+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 >> v1:172.21.15.52:6789/0 conn(0x7f355c13e820 legacy=0x7f355c12ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.187+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 shutdown_connections 2024-03-29T11:29:34.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.187+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 >> 172.21.15.52:0/2044075111 conn(0x7f355c06b660 msgr2=0x7f355c14d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:34.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.189+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 shutdown_connections 2024-03-29T11:29:34.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.191+0000 7f3560e64640 1 -- 172.21.15.52:0/2044075111 wait complete. 2024-03-29T11:29:34.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:34.191+0000 7f3560e64640 1 librados: shutdown 2024-03-29T11:29:34.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:41.007 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.164+0000 7fc798d1c640 10 monclient: get_monmap_and_config 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.164+0000 7fc798d1c640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.164+0000 7fc798d1c640 10 monclient: monmap: 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:41.165599+0000 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:41.165599+0000 2024-03-29T11:29:41.156 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.166+0000 7fc798d1c640 1 Processor -- start 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.166+0000 7fc798d1c640 1 -- start start 2024-03-29T11:29:41.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.166+0000 7fc798d1c640 10 monclient: init 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fc79406c250 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fc794072400 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fc79406c630 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79406d760 con 0x7fc79406c250 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc794070500 con 0x7fc79406c630 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940aff50 con 0x7fc794072400 2024-03-29T11:29:41.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.167+0000 7fc798d1c640 10 monclient(hunting): authenticate will time out at 1511.307895s 2024-03-29T11:29:41.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.169+0000 7fc79259c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc79406c250 0x7fc794071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33502/0 (socket says 172.21.15.52:33502) 2024-03-29T11:29:41.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.169+0000 7fc79259c640 1 -- 172.21.15.52:0/4249134355 learned_addr learned my addr 172.21.15.52:0/4249134355 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:41.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82184609 0 0) 0x7fc79406d760 con 0x7fc79406c250 2024-03-29T11:29:41.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:29:41.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77c003520 con 0x7fc79406c250 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255012065 0 0) 0x7fc7940aff50 con 0x7fc794072400 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.171+0000 7fc79159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc79406d760 con 0x7fc794072400 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4109307734 0 0) 0x7fc77c003520 con 0x7fc79406c250 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 >> v1:172.21.15.52:6790/0 conn(0x7fc794072400 legacy=0x7fc7941465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 >> v1:172.21.15.79:6789/0 conn(0x7fc79406c630 legacy=0x7fc794151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 1 monclient: found mon.noname-a 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.172+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940b00e0 con 0x7fc79406c250 2024-03-29T11:29:41.166 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc798d1c640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc798d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc788001690 con 0x7fc79406c250 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc79159a640 10 monclient: dump: 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:41.167 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:41.168 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:41.168 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:41.168 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:41.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.173+0000 7fc798d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.175+0000 7fc79159a640 1 -- 172.21.15.52:0/4249134355 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7880018b0 con 0x7fc79406c250 2024-03-29T11:29:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.176+0000 7fc79159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.176+0000 7fc798d1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.176+0000 7fc798d1c640 4 set_mon_vals no callback set 2024-03-29T11:29:41.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.177+0000 7fc798d1c640 10 monclient: shutdown 2024-03-29T11:29:41.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.177+0000 7fc798d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.177+0000 7fc798d1c640 1 -- 172.21.15.52:0/4249134355 >> v1:172.21.15.52:6789/0 conn(0x7fc79406c250 legacy=0x7fc794071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.177+0000 7fc798d1c640 1 -- 172.21.15.52:0/4249134355 shutdown_connections 2024-03-29T11:29:41.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.177+0000 7fc798d1c640 1 -- 172.21.15.52:0/4249134355 >> 172.21.15.52:0/4249134355 conn(0x7fc794067610 msgr2=0x7fc7940b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.179+0000 7fc798d1c640 1 -- 172.21.15.52:0/4249134355 shutdown_connections 2024-03-29T11:29:41.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.180+0000 7fc798d1c640 1 -- 172.21.15.52:0/4249134355 wait complete. 2024-03-29T11:29:41.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.181+0000 7fc798d1c640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.181+0000 7fc798d1c640 10 monclient: monmap: 2024-03-29T11:29:41.173 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.181+0000 7fc798d1c640 1 librados: starting msgr at 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.181+0000 7fc798d1c640 1 librados: starting objecter 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 1 Processor -- start 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 1 -- start start 2024-03-29T11:29:41.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 1 librados: setting wanted keys 2024-03-29T11:29:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 1 librados: calling monclient init 2024-03-29T11:29:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 10 monclient: init 2024-03-29T11:29:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:41.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.182+0000 7fc798d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fc79406c250 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:41.176 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.183+0000 7fc798d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fc79406c630 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.184+0000 7fc791d9b640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc79406c630 0x7fc7941458b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:35376/0 (socket says 172.21.15.52:35376) 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc798d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fc794072400 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc791d9b640 1 -- 172.21.15.52:0/151445798 learned_addr learned my addr 172.21.15.52:0/151445798 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940ab3d0 con 0x7fc79406c630 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940aaf40 con 0x7fc79406c250 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7941e8860 con 0x7fc794072400 2024-03-29T11:29:41.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc798d1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc798d1c640 10 monclient(hunting): authenticate will time out at 1511.325456s 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc782ffd640 10 client.?.objecter ms_handle_connect 0x7fc79406c630 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.185+0000 7fc782ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981448523 0 0) 0x7fc7940ab3d0 con 0x7fc79406c630 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc774004eb0 con 0x7fc79406c630 2024-03-29T11:29:41.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 client.?.objecter ms_handle_connect 0x7fc79406c250 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3894805570 0 0) 0x7fc774004eb0 con 0x7fc79406c630 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7940ab3d0 con 0x7fc79406c630 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc78c001cf0 con 0x7fc79406c630 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:41.179 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899832396 0 0) 0x7fc7940aaf40 con 0x7fc79406c250 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc774004eb0 con 0x7fc79406c250 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1477034380 0 0) 0x7fc7940ab3d0 con 0x7fc79406c630 2024-03-29T11:29:41.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.186+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 >> v1:172.21.15.52:6790/0 conn(0x7fc794072400 legacy=0x7fc7941e8150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 >> v1:172.21.15.79:6789/0 conn(0x7fc79406c250 legacy=0x7fc7941e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 1 monclient: found mon.a 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940ac0e0 con 0x7fc79406c630 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc78c001fd0 con 0x7fc79406c630 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fc78c003de0 con 0x7fc79406c630 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.188+0000 7fc782ffd640 10 monclient: dump: 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:41.182 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 10 monclient: _renew_subs 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7940ac3c0 con 0x7fc79406c630 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 10 monclient: _renew_subs 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7940a94e0 con 0x7fc79406c630 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.189+0000 7fc798d1c640 1 librados: init done 2024-03-29T11:29:41.183 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.190+0000 7fc783fff640 4 set_mon_vals no callback set 2024-03-29T11:29:41.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.192+0000 7fc798d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.193+0000 7fc798d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.193+0000 7fc798d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:41.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.193+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc778005080 con 0x7fc79406c630 2024-03-29T11:29:41.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.195+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fc78c003f40 con 0x7fc79406c630 2024-03-29T11:29:41.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.197+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7fc78c084dc0 con 0x7fc79406c630 2024-03-29T11:29:41.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.197+0000 7fc782ffd640 10 client.4554.objecter ms_dispatch 0x7fc794000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:41.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.197+0000 7fc782ffd640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:41.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.197+0000 7fc782ffd640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:41.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.197+0000 7fc782ffd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:29:41.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.198+0000 7fc79259c640 10 monclient: get_auth_request con 0x7fc77406dbd0 auth_method 0 2024-03-29T11:29:41.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.199+0000 7fc782ffd640 10 client.4554.objecter ms_handle_connect 0x7fc77406dbd0 2024-03-29T11:29:41.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.203+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc78c0506a0 con 0x7fc79406c630 2024-03-29T11:29:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.203+0000 7fc782ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.203+0000 7fc782ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:41.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.531+0000 7fc798d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.531+0000 7fc798d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.531+0000 7fc798d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:41.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.531+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc778005330 con 0x7fc79406c630 2024-03-29T11:29:41.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.533+0000 7fc782ffd640 1 -- 172.21.15.52:0/151445798 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7fc78c0525d0 con 0x7fc79406c630 2024-03-29T11:29:41.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.534+0000 7fc782ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.534+0000 7fc782ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:41.526 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:41.526 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:41.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.537+0000 7fc798d1c640 10 librados: watch_flush enter 2024-03-29T11:29:41.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.537+0000 7fc798d1c640 10 librados: watch_flush exit 2024-03-29T11:29:41.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.537+0000 7fc798d1c640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.537+0000 7fc798d1c640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.537+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 >> v1:172.21.15.79:6816/407480584 conn(0x7fc77406dbd0 legacy=0x7fc774070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.537+0000 7fc798d1c640 10 monclient: shutdown 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.538+0000 7fc798d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.538+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 >> v1:172.21.15.52:6789/0 conn(0x7fc79406c630 legacy=0x7fc7941458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.538+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 shutdown_connections 2024-03-29T11:29:41.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.538+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 >> 172.21.15.52:0/151445798 conn(0x7fc794067610 msgr2=0x7fc794070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.538+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 shutdown_connections 2024-03-29T11:29:41.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.539+0000 7fc798d1c640 1 -- 172.21.15.52:0/151445798 wait complete. 2024-03-29T11:29:41.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:41.539+0000 7fc798d1c640 1 librados: shutdown 2024-03-29T11:29:41.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:48.355 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.509+0000 7efea2b23640 10 monclient: get_monmap_and_config 2024-03-29T11:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.509+0000 7efea2b23640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.509+0000 7efea2b23640 10 monclient: monmap: 2024-03-29T11:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:48.510536+0000 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:48.510536+0000 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:48.502 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.510+0000 7efea2b23640 1 Processor -- start 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.510+0000 7efea2b23640 1 -- start start 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.510+0000 7efea2b23640 10 monclient: init 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.511+0000 7efea2b23640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.511+0000 7efea2b23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.511+0000 7efea2b23640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c1457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.511+0000 7efea2b23640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c13f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.511+0000 7efea2b23640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c0a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.511+0000 7efea2b23640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0aaef0 con 0x7efe9c0a9d70 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efea10c0640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7efe9c0a9d70 0x7efe9c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:43058/0 (socket says 172.21.15.52:43058) 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9bfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7efe9c13f650 0x7efe9c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50612/0 (socket says 172.21.15.52:50612) 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9bfff640 1 -- 172.21.15.52:0/2846962032 learned_addr learned my addr 172.21.15.52:0/2846962032 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c145bd0 con 0x7efe9c1457f0 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c145eb0 con 0x7efe9c13f650 2024-03-29T11:29:48.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efea2b23640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efea2b23640 10 monclient(hunting): authenticate will time out at 1518.652606s 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296848915 0 0) 0x7efe9c145bd0 con 0x7efe9c1457f0 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88003520 con 0x7efe9c1457f0 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367184464 0 0) 0x7efe9c145eb0 con 0x7efe9c13f650 2024-03-29T11:29:48.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.512+0000 7efe9b7fe640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c145bd0 con 0x7efe9c13f650 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581544930 0 0) 0x7efe9c145bd0 con 0x7efe9c13f650 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 >> v1:172.21.15.79:6789/0 conn(0x7efe9c1457f0 legacy=0x7efe9c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 >> v1:172.21.15.52:6789/0 conn(0x7efe9c0a9d70 legacy=0x7efe9c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c146340 con 0x7efe9c13f650 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efe8c0019f0 con 0x7efe9c13f650 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: dump: 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efea2b23640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efea2b23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe8c001b50 con 0x7efe9c13f650 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 1 -- 172.21.15.52:0/2846962032 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efe8c001fa0 con 0x7efe9c13f650 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efea2b23640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efea2b23640 4 set_mon_vals no callback set 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.513+0000 7efe9b7fe640 10 monclient: dump: 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.514+0000 7efea2b23640 10 monclient: shutdown 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.514+0000 7efea2b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.514+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 >> v1:172.21.15.52:6790/0 conn(0x7efe9c13f650 legacy=0x7efe9c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.514+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 shutdown_connections 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.514+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 >> 172.21.15.52:0/2846962032 conn(0x7efe9c066a00 msgr2=0x7efe9c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.515+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 shutdown_connections 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.515+0000 7efea2b23640 1 -- 172.21.15.52:0/2846962032 wait complete. 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.516+0000 7efea2b23640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.516+0000 7efea2b23640 10 monclient: monmap: 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.516+0000 7efea2b23640 1 librados: starting msgr at 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.517+0000 7efea2b23640 1 librados: starting objecter 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.517+0000 7efea2b23640 1 Processor -- start 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.517+0000 7efea2b23640 1 -- start start 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.517+0000 7efea2b23640 1 librados: setting wanted keys 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.517+0000 7efea2b23640 1 librados: calling monclient init 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.517+0000 7efea2b23640 10 monclient: init 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c13f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c1457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c142970 con 0x7efe9c1457f0 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c1daa80 con 0x7efe9c13f650 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c13b900 con 0x7efe9c0a9d70 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.518+0000 7efea2b23640 10 monclient(hunting): authenticate will time out at 1518.658826s 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe9bfff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7efe9c13f650 0x7efe9c142260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:33452/0 (socket says 172.21.15.52:33452) 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe9bfff640 1 -- 172.21.15.52:0/2752429606 learned_addr learned my addr 172.21.15.52:0/2752429606 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 client.?.objecter ms_handle_connect 0x7efe9c0a9d70 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 client.?.objecter ms_handle_connect 0x7efe9c1457f0 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535102642 0 0) 0x7efe9c13b900 con 0x7efe9c0a9d70 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:29:48.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.519+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe7c004eb0 con 0x7efe9c0a9d70 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654203698 0 0) 0x7efe9c142970 con 0x7efe9c1457f0 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c13b900 con 0x7efe9c1457f0 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145319043 0 0) 0x7efe7c004eb0 con 0x7efe9c0a9d70 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9c142970 con 0x7efe9c0a9d70 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efe94001890 con 0x7efe9c0a9d70 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): dump: 2024-03-29T11:29:48.524 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092868009 0 0) 0x7efe9c13b900 con 0x7efe9c1457f0 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe7c004eb0 con 0x7efe9c1457f0 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efe84001830 con 0x7efe9c1457f0 2024-03-29T11:29:48.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 monclient(hunting): dump: 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 client.?.objecter ms_handle_connect 0x7efe9c13f650 2024-03-29T11:29:48.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.520+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 421956095 0 0) 0x7efe7c004eb0 con 0x7efe9c1457f0 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 >> v1:172.21.15.52:6790/0 conn(0x7efe9c0a9d70 legacy=0x7efe9c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 >> v1:172.21.15.79:6789/0 conn(0x7efe9c13f650 legacy=0x7efe9c142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 1 monclient: found mon.a 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c1d4ae0 con 0x7efe9c1457f0 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 10 monclient: _renew_subs 2024-03-29T11:29:48.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c13b4b0 con 0x7efe9c1457f0 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe84001d40 con 0x7efe9c1457f0 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 10 monclient: _renew_subs 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c1da6f0 con 0x7efe9c1457f0 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efea2b23640 1 librados: init done 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.521+0000 7efe9a7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.522+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7efe84004980 con 0x7efe9c1457f0 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.522+0000 7efe997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.522+0000 7efe997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.522+0000 7efe997fa640 10 monclient: dump: 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:48.528 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.523+0000 7efea2b23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.523+0000 7efea2b23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.523+0000 7efea2b23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.523+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80005080 con 0x7efe9c1457f0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.524+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7efe8401a950 con 0x7efe9c1457f0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7efe84084860 con 0x7efe9c1457f0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efe997fa640 10 client.4560.objecter ms_dispatch 0x7efe9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efe997fa640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efe997fa640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efea08bf640 10 monclient: get_auth_request con 0x7efe7c06de20 auth_method 0 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efe997fa640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.525+0000 7efe997fa640 10 client.4560.objecter ms_handle_connect 0x7efe7c06de20 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.534+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe840501a0 con 0x7efe9c1457f0 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.534+0000 7efe997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.534+0000 7efe997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.924+0000 7efea2b23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.924+0000 7efea2b23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.924+0000 7efea2b23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:29:48.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.924+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe80005330 con 0x7efe9c1457f0 2024-03-29T11:29:48.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.926+0000 7efe997fa640 1 -- 172.21.15.52:0/2752429606 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7efe840520d0 con 0x7efe9c1457f0 2024-03-29T11:29:48.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.926+0000 7efe997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.926+0000 7efe997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:48.919 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:48.919 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 10 librados: watch_flush enter 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 10 librados: watch_flush exit 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 >> v1:172.21.15.79:6816/407480584 conn(0x7efe7c06de20 legacy=0x7efe7c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 10 monclient: shutdown 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.930+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 >> v1:172.21.15.52:6789/0 conn(0x7efe9c1457f0 legacy=0x7efe9c1d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.931+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 shutdown_connections 2024-03-29T11:29:48.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.931+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 >> 172.21.15.52:0/2752429606 conn(0x7efe9c066a00 msgr2=0x7efe9c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.931+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 shutdown_connections 2024-03-29T11:29:48.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.931+0000 7efea2b23640 1 -- 172.21.15.52:0/2752429606 wait complete. 2024-03-29T11:29:48.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:48.931+0000 7efea2b23640 1 librados: shutdown 2024-03-29T11:29:48.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:55.746 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:55.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.890+0000 7f735c36b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:55.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.890+0000 7f735c36b640 10 monclient: build_initial_monmap 2024-03-29T11:29:55.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.890+0000 7f735c36b640 10 monclient: monmap: 2024-03-29T11:29:55.883 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:55.883 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:55.883 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:29:55.892083+0000 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:29:55.892083+0000 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.890+0000 7f735c36b640 1 Processor -- start 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.890+0000 7f735c36b640 1 -- start start 2024-03-29T11:29:55.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.890+0000 7f735c36b640 10 monclient: init 2024-03-29T11:29:55.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:55.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 10 monclient(hunting): picked mon.noname-b con 0x7f73541457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 10 monclient(hunting): picked mon.noname-c con 0x7f735413f650 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 10 monclient(hunting): picked mon.noname-a con 0x7f73540a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540aaef0 con 0x7f73540a9d70 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.891+0000 7f735c36b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7354145bd0 con 0x7f735413f650 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.892+0000 7f7359906640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f735413f650 0x7f73540b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:57634/0 (socket says 172.21.15.52:57634) 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.892+0000 7f7359906640 1 -- 172.21.15.52:0/196656188 learned_addr learned my addr 172.21.15.52:0/196656188 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.892+0000 7f735c36b640 1 -- 172.21.15.52:0/196656188 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7354145eb0 con 0x7f73541457f0 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.892+0000 7f735c36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.892+0000 7f735c36b640 10 monclient(hunting): authenticate will time out at 1526.033806s 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735269514 0 0) 0x7f7354145eb0 con 0x7f73541457f0 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7340003520 con 0x7f73541457f0 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378574155 0 0) 0x7f7340003520 con 0x7f73541457f0 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 >> v1:172.21.15.79:6789/0 conn(0x7f735413f650 legacy=0x7f73540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.893+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 >> v1:172.21.15.52:6789/0 conn(0x7f73540a9d70 legacy=0x7f735414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: _finish_hunting 0 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 1 monclient: found mon.noname-b 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:55.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354146340 con 0x7f73541457f0 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: _finish_auth 0 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: _check_auth_tickets 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7350001690 con 0x7f73541457f0 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f7359105640 10 monclient: dump: 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:55.889 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:55.890 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:55.890 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:55.890 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:55.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f735c36b640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:29:55.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.895+0000 7f735c36b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7350001860 con 0x7f73541457f0 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f7359105640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f735c36b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f735c36b640 4 set_mon_vals no callback set 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f7359105640 1 -- 172.21.15.52:0/196656188 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7350001cb0 con 0x7f73541457f0 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f7359105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f7359105640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.896+0000 7f7359105640 10 monclient: dump: 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.892 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 10 monclient: shutdown 2024-03-29T11:29:55.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:55.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 1 -- 172.21.15.52:0/196656188 >> v1:172.21.15.52:6790/0 conn(0x7f73541457f0 legacy=0x7f735413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 1 -- 172.21.15.52:0/196656188 shutdown_connections 2024-03-29T11:29:55.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 1 -- 172.21.15.52:0/196656188 >> 172.21.15.52:0/196656188 conn(0x7f7354066a00 msgr2=0x7f7354066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 1 -- 172.21.15.52:0/196656188 shutdown_connections 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.897+0000 7f735c36b640 1 -- 172.21.15.52:0/196656188 wait complete. 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.898+0000 7f735c36b640 10 monclient: build_initial_monmap 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.898+0000 7f735c36b640 10 monclient: monmap: 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:29:55.895 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 1 librados: starting msgr at 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 1 librados: starting objecter 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 1 Processor -- start 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 1 -- start start 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 1 librados: setting wanted keys 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 1 librados: calling monclient init 2024-03-29T11:29:55.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.899+0000 7f735c36b640 10 monclient: init 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 10 monclient(hunting): picked mon.noname-b con 0x7f73540a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 10 monclient(hunting): picked mon.noname-a con 0x7f735413f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 10 monclient(hunting): picked mon.noname-c con 0x7f73541457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540af620 con 0x7f735413f650 2024-03-29T11:29:55.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540adf10 con 0x7f73540a9d70 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540ae430 con 0x7f73541457f0 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.900+0000 7f735c36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.901+0000 7f735c36b640 10 monclient(hunting): authenticate will time out at 1526.042365s 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.901+0000 7f735a107640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f73540a9d70 0x7f73541da8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:57768/0 (socket says 172.21.15.52:57768) 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.901+0000 7f735a107640 1 -- 172.21.15.52:0/1684005964 learned_addr learned my addr 172.21.15.52:0/1684005964 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 client.?.objecter ms_handle_connect 0x7f73540a9d70 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 client.?.objecter ms_handle_connect 0x7f73541457f0 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605037365 0 0) 0x7f73540ae430 con 0x7f73541457f0 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338004eb0 con 0x7f73541457f0 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2819834978 0 0) 0x7f7338004eb0 con 0x7f73541457f0 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73540ae430 con 0x7f73541457f0 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f734c001aa0 con 0x7f73541457f0 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): dump: 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:55.903 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 907259425 0 0) 0x7f73540ae430 con 0x7f73541457f0 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 >> v1:172.21.15.79:6789/0 conn(0x7f73540a9d70 legacy=0x7f73541da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.902+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 >> v1:172.21.15.52:6789/0 conn(0x7f735413f650 legacy=0x7f73540b0d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 1 monclient: found mon.c 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540adaf0 con 0x7f73541457f0 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:55.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.903+0000 7f734affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 10 monclient: _renew_subs 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73541db000 con 0x7f73541457f0 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 10 monclient: _renew_subs 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7354142410 con 0x7f73541457f0 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.904+0000 7f735c36b640 1 librados: init done 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.905+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f734c002030 con 0x7f73541457f0 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.905+0000 7f734affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.905+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f734c003b10 con 0x7f73541457f0 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.905+0000 7f734affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.905+0000 7f734affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:55.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.905+0000 7f734affd640 10 monclient: dump: 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.906+0000 7f735c36b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.906+0000 7f735c36b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.906+0000 7f735c36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:55.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.906+0000 7f7358904640 4 set_mon_vals no callback set 2024-03-29T11:29:55.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.906+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7354075f10 con 0x7f73541457f0 2024-03-29T11:29:55.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.908+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f734c019ae0 con 0x7f73541457f0 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.908+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3433376428 0 0) 0x7f734c083a10 con 0x7f73541457f0 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.908+0000 7f734affd640 10 client.4631.objecter ms_dispatch 0x7f7354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.908+0000 7f734affd640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.908+0000 7f734affd640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.909+0000 7f734affd640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.909+0000 7f735a107640 10 monclient: get_auth_request con 0x7f733806db80 auth_method 0 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.909+0000 7f734affd640 10 client.4631.objecter ms_handle_connect 0x7f733806db80 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.914+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f734c04f120 con 0x7f73541457f0 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.914+0000 7f734affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:55.914+0000 7f734affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:56.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.241+0000 7f735c36b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.241+0000 7f735c36b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.241+0000 7f735c36b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:29:56.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.241+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73540b3680 con 0x7f73541457f0 2024-03-29T11:29:56.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.242+0000 7f734affd640 1 -- 172.21.15.52:0/1684005964 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 689003469) 0x7f734c051220 con 0x7f73541457f0 2024-03-29T11:29:56.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.242+0000 7f734affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.242+0000 7f734affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:56.236 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:29:56.236 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:19:59.258675+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:59.258675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:56.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 10 librados: watch_flush enter 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 10 librados: watch_flush exit 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 >> v1:172.21.15.79:6816/407480584 conn(0x7f733806db80 legacy=0x7f7338070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 10 monclient: shutdown 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 >> v1:172.21.15.52:6790/0 conn(0x7f73541457f0 legacy=0x7f73541e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 shutdown_connections 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 >> 172.21.15.52:0/1684005964 conn(0x7f7354066a00 msgr2=0x7f7354066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 shutdown_connections 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 -- 172.21.15.52:0/1684005964 wait complete. 2024-03-29T11:29:56.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:29:56.245+0000 7f735c36b640 1 librados: shutdown 2024-03-29T11:29:56.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:03.060 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:03.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.205+0000 7fa3c9ecf640 10 monclient: get_monmap_and_config 2024-03-29T11:30:03.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.205+0000 7fa3c9ecf640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.197 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.205+0000 7fa3c9ecf640 10 monclient: monmap: 2024-03-29T11:30:03.197 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:03.206209+0000 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:03.206209+0000 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:03.198 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:03.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.206+0000 7fa3c9ecf640 1 Processor -- start 2024-03-29T11:30:03.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.206+0000 7fa3c9ecf640 1 -- start start 2024-03-29T11:30:03.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.206+0000 7fa3c9ecf640 10 monclient: init 2024-03-29T11:30:03.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:03.199 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c41457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c413f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c40a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c40aaef0 con 0x7fa3c40a9d70 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4145bd0 con 0x7fa3c41457f0 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4145eb0 con 0x7fa3c413f650 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.207+0000 7fa3c9ecf640 10 monclient(hunting): authenticate will time out at 1533.347926s 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3baffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa3c413f650 0x7fa3c40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:35368/0 (socket says 172.21.15.52:35368) 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3baffd640 1 -- 172.21.15.52:0/3119816615 learned_addr learned my addr 172.21.15.52:0/3119816615 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023537270 0 0) 0x7fa3c4145eb0 con 0x7fa3c413f650 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b0003520 con 0x7fa3c413f650 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059442026 0 0) 0x7fa3b0003520 con 0x7fa3c413f650 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 >> v1:172.21.15.79:6789/0 conn(0x7fa3c41457f0 legacy=0x7fa3c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:03.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 >> v1:172.21.15.52:6789/0 conn(0x7fa3c40a9d70 legacy=0x7fa3c414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4146340 con 0x7fa3c413f650 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa3ac001690 con 0x7fa3c413f650 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: dump: 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:03.204 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c9ecf640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:30:03.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c9ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3ac001860 con 0x7fa3c413f650 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c9ecf640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c9ecf640 4 set_mon_vals no callback set 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 1 -- 172.21.15.52:0/3119816615 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa3ac001cb0 con 0x7fa3c413f650 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.208+0000 7fa3c2ffd640 10 monclient: dump: 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.206 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 10 monclient: shutdown 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/3119816615 >> v1:172.21.15.52:6790/0 conn(0x7fa3c413f650 legacy=0x7fa3c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/3119816615 shutdown_connections 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/3119816615 >> 172.21.15.52:0/3119816615 conn(0x7fa3c4066a00 msgr2=0x7fa3c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/3119816615 shutdown_connections 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.209+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/3119816615 wait complete. 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.210+0000 7fa3c9ecf640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.210+0000 7fa3c9ecf640 10 monclient: monmap: 2024-03-29T11:30:03.207 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.210+0000 7fa3c9ecf640 1 librados: starting msgr at 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.211+0000 7fa3c9ecf640 1 librados: starting objecter 2024-03-29T11:30:03.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.211+0000 7fa3c9ecf640 1 Processor -- start 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.211+0000 7fa3c9ecf640 1 -- start start 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.211+0000 7fa3c9ecf640 1 librados: setting wanted keys 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.211+0000 7fa3c9ecf640 1 librados: calling monclient init 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.211+0000 7fa3c9ecf640 10 monclient: init 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.212+0000 7fa3c9ecf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.212+0000 7fa3c9ecf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.212+0000 7fa3c9ecf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c40a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:03.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.212+0000 7fa3c9ecf640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c413f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.212+0000 7fa3baffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa3c413f650 0x7fa3c4143320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:35842/0 (socket says 172.21.15.52:35842) 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c9ecf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c41457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3baffd640 1 -- 172.21.15.52:0/2646586992 learned_addr learned my addr 172.21.15.52:0/2646586992 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4140350 con 0x7fa3c41457f0 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c4145020 con 0x7fa3c40a9d70 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c41451b0 con 0x7fa3c413f650 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c9ecf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c9ecf640 10 monclient(hunting): authenticate will time out at 1533.353292s 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3c413f650 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349513349 0 0) 0x7fa3c41451b0 con 0x7fa3c413f650 2024-03-29T11:30:03.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.213+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a4004eb0 con 0x7fa3c413f650 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3c40a9d70 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428047665 0 0) 0x7fa3a4004eb0 con 0x7fa3c413f650 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3c41451b0 con 0x7fa3c413f650 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa3ac001b70 con 0x7fa3c413f650 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:03.215 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.214+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761793786 0 0) 0x7fa3c4145020 con 0x7fa3c40a9d70 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a4004eb0 con 0x7fa3c40a9d70 2024-03-29T11:30:03.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2762042146 0 0) 0x7fa3c41451b0 con 0x7fa3c413f650 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 >> v1:172.21.15.79:6789/0 conn(0x7fa3c40a9d70 legacy=0x7fa3c4134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 >> v1:172.21.15.52:6789/0 conn(0x7fa3c41457f0 legacy=0x7fa3c4143a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 1 monclient: found mon.c 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4145340 con 0x7fa3c413f650 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 10 monclient: _renew_subs 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c413fe00 con 0x7fa3c413f650 2024-03-29T11:30:03.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 10 monclient: _renew_subs 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c413dd90 con 0x7fa3c413f650 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.215+0000 7fa3c9ecf640 1 librados: init done 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.217+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3ac004bf0 con 0x7fa3c413f650 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.217+0000 7fa3c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.217+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fa3ac004f00 con 0x7fa3c413f650 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.217+0000 7fa3c9ecf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.217+0000 7fa3c1ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c9ecf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c9ecf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:03.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c4140590 con 0x7fa3c413f650 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c0ff9640 10 monclient: dump: 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:03.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.218+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fa3ac01af90 con 0x7fa3c413f650 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c37fe640 10 monclient: get_auth_request con 0x7fa3a406ddd0 auth_method 0 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7fa3ac084cf0 con 0x7fa3c413f650 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c0ff9640 10 client.4637.objecter ms_dispatch 0x7fa3c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c0ff9640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c0ff9640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c0ff9640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.219+0000 7fa3c0ff9640 10 client.4637.objecter ms_handle_connect 0x7fa3a406ddd0 2024-03-29T11:30:03.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.225+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3c4140590 con 0x7fa3c413f650 2024-03-29T11:30:03.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.225+0000 7fa3c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.225+0000 7fa3c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:03.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.543+0000 7fa3c9ecf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.543+0000 7fa3c9ecf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.543+0000 7fa3c9ecf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:03.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.543+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c4140590 con 0x7fa3c413f650 2024-03-29T11:30:03.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.544+0000 7fa3c0ff9640 1 -- 172.21.15.52:0/2646586992 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7fa3c4140590 con 0x7fa3c413f650 2024-03-29T11:30:03.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.544+0000 7fa3c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.544+0000 7fa3c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:03.537 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:03.537 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:03.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 10 librados: watch_flush enter 2024-03-29T11:30:03.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 10 librados: watch_flush exit 2024-03-29T11:30:03.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:30:03.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 >> v1:172.21.15.79:6816/407480584 conn(0x7fa3a406ddd0 legacy=0x7fa3a4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 10 monclient: shutdown 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 >> v1:172.21.15.52:6790/0 conn(0x7fa3c413f650 legacy=0x7fa3c4143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 shutdown_connections 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 >> 172.21.15.52:0/2646586992 conn(0x7fa3c4066a00 msgr2=0x7fa3c40b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.548+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 shutdown_connections 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.549+0000 7fa3c9ecf640 1 -- 172.21.15.52:0/2646586992 wait complete. 2024-03-29T11:30:03.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:03.549+0000 7fa3c9ecf640 1 librados: shutdown 2024-03-29T11:30:03.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:10.365 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:10.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.527+0000 7fe19f3c4640 10 monclient: get_monmap_and_config 2024-03-29T11:30:10.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.527+0000 7fe19f3c4640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.528+0000 7fe19f3c4640 10 monclient: monmap: 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:10.528943+0000 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:10.528943+0000 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:10.521 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:10.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.528+0000 7fe19f3c4640 1 Processor -- start 2024-03-29T11:30:10.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.529+0000 7fe19f3c4640 1 -- start start 2024-03-29T11:30:10.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.529+0000 7fe19f3c4640 10 monclient: init 2024-03-29T11:30:10.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.530+0000 7fe19f3c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.530+0000 7fe19f3c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:10.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.530+0000 7fe19f3c4640 10 monclient(hunting): picked mon.noname-c con 0x7fe1981457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:10.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.530+0000 7fe19f3c4640 10 monclient(hunting): picked mon.noname-b con 0x7fe19813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:10.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.530+0000 7fe19f3c4640 10 monclient(hunting): picked mon.noname-a con 0x7fe1980a9d70 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:10.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.532+0000 7fe19f3c4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980aaef0 con 0x7fe1980a9d70 2024-03-29T11:30:10.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.532+0000 7fe19f3c4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe198145bd0 con 0x7fe1981457f0 2024-03-29T11:30:10.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.532+0000 7fe19f3c4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe198145eb0 con 0x7fe19813f650 2024-03-29T11:30:10.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.532+0000 7fe19f3c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.532+0000 7fe19f3c4640 10 monclient(hunting): authenticate will time out at 1540.672346s 2024-03-29T11:30:10.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe19c95f640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe19813f650 0x7fe1980b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:36514/0 (socket says 172.21.15.52:36514) 2024-03-29T11:30:10.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe19c95f640 1 -- 172.21.15.52:0/109263489 learned_addr learned my addr 172.21.15.52:0/109263489 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:10.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941218260 0 0) 0x7fe198145eb0 con 0x7fe19813f650 2024-03-29T11:30:10.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe18ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe18ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe18ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe18ffff640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:30:10.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.533+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe184003520 con 0x7fe19813f650 2024-03-29T11:30:10.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170334657 0 0) 0x7fe184003520 con 0x7fe19813f650 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 >> v1:172.21.15.79:6789/0 conn(0x7fe1981457f0 legacy=0x7fe19813ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 >> v1:172.21.15.52:6789/0 conn(0x7fe1980a9d70 legacy=0x7fe19814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 monclient: found mon.noname-b 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe198146340 con 0x7fe19813f650 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe188001690 con 0x7fe19813f650 2024-03-29T11:30:10.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: dump: 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe19f3c4640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe19f3c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:10.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe188001860 con 0x7fe19813f650 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 1 -- 172.21.15.52:0/109263489 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe188001c70 con 0x7fe19813f650 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.536+0000 7fe18ffff640 10 monclient: dump: 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:10.531 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:10.532 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.537+0000 7fe19f3c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.537+0000 7fe19f3c4640 4 set_mon_vals no callback set 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.538+0000 7fe19f3c4640 10 monclient: shutdown 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.538+0000 7fe19f3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.538+0000 7fe19f3c4640 1 -- 172.21.15.52:0/109263489 >> v1:172.21.15.52:6790/0 conn(0x7fe19813f650 legacy=0x7fe1980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.538+0000 7fe19f3c4640 1 -- 172.21.15.52:0/109263489 shutdown_connections 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.538+0000 7fe19f3c4640 1 -- 172.21.15.52:0/109263489 >> 172.21.15.52:0/109263489 conn(0x7fe198066a00 msgr2=0x7fe198066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.539+0000 7fe19f3c4640 1 -- 172.21.15.52:0/109263489 shutdown_connections 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.540+0000 7fe19f3c4640 1 -- 172.21.15.52:0/109263489 wait complete. 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.540+0000 7fe19f3c4640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.540+0000 7fe19f3c4640 10 monclient: monmap: 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:10.533 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:10.534 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 1 librados: starting msgr at 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 1 librados: starting objecter 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 1 Processor -- start 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 1 -- start start 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 1 librados: setting wanted keys 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 1 librados: calling monclient init 2024-03-29T11:30:10.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.542+0000 7fe19f3c4640 10 monclient: init 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 10 monclient(hunting): picked mon.noname-b con 0x7fe1980a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 10 monclient(hunting): picked mon.noname-a con 0x7fe19813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 10 monclient(hunting): picked mon.noname-c con 0x7fe1981457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980ad5c0 con 0x7fe19813f650 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19d961640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe1981457f0 0x7fe1981e2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37462/0 (socket says 172.21.15.52:37462) 2024-03-29T11:30:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19d961640 1 -- 172.21.15.52:0/2789125629 learned_addr learned my addr 172.21.15.52:0/2789125629 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.543+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980ad810 con 0x7fe1980a9d70 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.544+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980abd10 con 0x7fe1981457f0 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.546+0000 7fe19f3c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.546+0000 7fe19f3c4640 10 monclient(hunting): authenticate will time out at 1540.686818s 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.546+0000 7fe18dffb640 10 client.?.objecter ms_handle_connect 0x7fe1981457f0 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478091840 0 0) 0x7fe1980abd10 con 0x7fe1981457f0 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:30:10.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.547+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe17c004eb0 con 0x7fe1981457f0 2024-03-29T11:30:10.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653652979 0 0) 0x7fe17c004eb0 con 0x7fe1981457f0 2024-03-29T11:30:10.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1980abd10 con 0x7fe1981457f0 2024-03-29T11:30:10.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe190001d30 con 0x7fe1981457f0 2024-03-29T11:30:10.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 10 monclient(hunting): dump: 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.549+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 996221996 0 0) 0x7fe1980abd10 con 0x7fe1981457f0 2024-03-29T11:30:10.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 >> v1:172.21.15.79:6789/0 conn(0x7fe1980a9d70 legacy=0x7fe1980ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 >> v1:172.21.15.52:6789/0 conn(0x7fe19813f650 legacy=0x7fe198137cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:10.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 1 monclient: found mon.c 2024-03-29T11:30:10.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:10.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1980b0db0 con 0x7fe1981457f0 2024-03-29T11:30:10.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe18dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.550+0000 7fe19f3c4640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 10 monclient: _renew_subs 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1981db690 con 0x7fe1981457f0 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 10 monclient: _renew_subs 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1980b12e0 con 0x7fe1981457f0 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe190001e90 con 0x7fe1981457f0 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.551+0000 7fe19f3c4640 1 librados: init done 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.552+0000 7fe18dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.552+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fe190003de0 con 0x7fe1981457f0 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.552+0000 7fe18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.552+0000 7fe18dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.552+0000 7fe18dffb640 10 monclient: dump: 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:10.549 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.552+0000 7fe18f7fe640 4 set_mon_vals no callback set 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.553+0000 7fe19f3c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.554+0000 7fe19f3c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.554+0000 7fe19f3c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.554+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178005080 con 0x7fe1981457f0 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.554+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fe190003f40 con 0x7fe1981457f0 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.555+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7fe190084f20 con 0x7fe1981457f0 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.555+0000 7fe18dffb640 10 client.4643.objecter ms_dispatch 0x7fe198000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.555+0000 7fe19d160640 10 monclient: get_auth_request con 0x7fe17c06dc60 auth_method 0 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.556+0000 7fe18dffb640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:10.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.556+0000 7fe18dffb640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:10.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.556+0000 7fe18dffb640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:30:10.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.556+0000 7fe18dffb640 10 client.4643.objecter ms_handle_connect 0x7fe17c06dc60 2024-03-29T11:30:10.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.563+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe190050800 con 0x7fe1981457f0 2024-03-29T11:30:10.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.563+0000 7fe18dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.563+0000 7fe18dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:10.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.900+0000 7fe19f3c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.900+0000 7fe19f3c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.900+0000 7fe19f3c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.900+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe178005330 con 0x7fe1981457f0 2024-03-29T11:30:10.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.901+0000 7fe18dffb640 1 -- 172.21.15.52:0/2789125629 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7fe190052730 con 0x7fe1981457f0 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.901+0000 7fe18dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.901+0000 7fe18dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:10.894 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:10.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.904+0000 7fe19f3c4640 10 librados: watch_flush enter 2024-03-29T11:30:10.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.904+0000 7fe19f3c4640 10 librados: watch_flush exit 2024-03-29T11:30:10.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.904+0000 7fe19f3c4640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:30:10.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.904+0000 7fe19f3c4640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:30:10.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.905+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 >> v1:172.21.15.79:6816/407480584 conn(0x7fe17c06dc60 legacy=0x7fe17c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.905+0000 7fe19f3c4640 10 monclient: shutdown 2024-03-29T11:30:10.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.905+0000 7fe19f3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.905+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 >> v1:172.21.15.52:6790/0 conn(0x7fe1981457f0 legacy=0x7fe1981e2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.906+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 shutdown_connections 2024-03-29T11:30:10.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.906+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 >> 172.21.15.52:0/2789125629 conn(0x7fe198066a00 msgr2=0x7fe1980b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.906+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 shutdown_connections 2024-03-29T11:30:10.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.906+0000 7fe19f3c4640 1 -- 172.21.15.52:0/2789125629 wait complete. 2024-03-29T11:30:10.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:10.906+0000 7fe19f3c4640 1 librados: shutdown 2024-03-29T11:30:10.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:17.722 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:17.871 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.877+0000 7f9ac9be6640 10 monclient: get_monmap_and_config 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.877+0000 7f9ac9be6640 10 monclient: build_initial_monmap 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.877+0000 7f9ac9be6640 10 monclient: monmap: 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:17.878663+0000 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:17.878663+0000 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.879+0000 7f9ac9be6640 1 Processor -- start 2024-03-29T11:30:17.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.879+0000 7f9ac9be6640 1 -- start start 2024-03-29T11:30:17.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.879+0000 7f9ac9be6640 10 monclient: init 2024-03-29T11:30:17.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.880+0000 7f9ac9be6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:17.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.880+0000 7f9ac9be6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac40a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac4149ac0 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac414fe50 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac40aa4a0 con 0x7f9ac40a9750 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac414a680 con 0x7f9ac414fe50 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac40af290 con 0x7f9ac4149ac0 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:17.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.881+0000 7f9ac9be6640 10 monclient(hunting): authenticate will time out at 1548.021729s 2024-03-29T11:30:17.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.884+0000 7f9abbfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9ac4149ac0 0x7f9ac4149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:36438/0 (socket says 172.21.15.52:36438) 2024-03-29T11:30:17.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.884+0000 7f9abbfff640 1 -- 172.21.15.52:0/3755257977 learned_addr learned my addr 172.21.15.52:0/3755257977 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:17.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.886+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164349790 0 0) 0x7f9ac40af290 con 0x7f9ac4149ac0 2024-03-29T11:30:17.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.886+0000 7f9ac2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.886+0000 7f9ac2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.886+0000 7f9ac2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.886+0000 7f9ac2ffd640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.886+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab0003520 con 0x7f9ac4149ac0 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944199706 0 0) 0x7f9ab0003520 con 0x7f9ac4149ac0 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 >> v1:172.21.15.79:6789/0 conn(0x7f9ac414fe50 legacy=0x7f9ac4152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 >> v1:172.21.15.52:6789/0 conn(0x7f9ac40a9750 legacy=0x7f9ac41436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:30:17.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac40afb40 con 0x7f9ac4149ac0 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9aac001690 con 0x7f9ac4149ac0 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac9be6640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac9be6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac2ffd640 10 monclient: dump: 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.880 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.881 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:17.881 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:17.881 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:17.881 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.887+0000 7f9ac9be6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aac0017f0 con 0x7f9ac4149ac0 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac2ffd640 1 -- 172.21.15.52:0/3755257977 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9aac001c40 con 0x7f9ac4149ac0 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac9be6640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac9be6640 4 set_mon_vals no callback set 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.888+0000 7f9ac2ffd640 10 monclient: dump: 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.882 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.889+0000 7f9ac9be6640 10 monclient: shutdown 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.889+0000 7f9ac9be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.889+0000 7f9ac9be6640 1 -- 172.21.15.52:0/3755257977 >> v1:172.21.15.52:6790/0 conn(0x7f9ac4149ac0 legacy=0x7f9ac4149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.891+0000 7f9ac9be6640 1 -- 172.21.15.52:0/3755257977 shutdown_connections 2024-03-29T11:30:17.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.891+0000 7f9ac9be6640 1 -- 172.21.15.52:0/3755257977 >> 172.21.15.52:0/3755257977 conn(0x7f9ac40b35e0 msgr2=0x7f9ac40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:17.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.892+0000 7f9ac9be6640 1 -- 172.21.15.52:0/3755257977 shutdown_connections 2024-03-29T11:30:17.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.892+0000 7f9ac9be6640 1 -- 172.21.15.52:0/3755257977 wait complete. 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.893+0000 7f9ac9be6640 10 monclient: build_initial_monmap 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.893+0000 7f9ac9be6640 10 monclient: monmap: 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:17.886 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.893+0000 7f9ac9be6640 1 librados: starting msgr at 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.893+0000 7f9ac9be6640 1 librados: starting objecter 2024-03-29T11:30:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.894+0000 7f9ac9be6640 1 Processor -- start 2024-03-29T11:30:17.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.895+0000 7f9ac9be6640 1 -- start start 2024-03-29T11:30:17.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.895+0000 7f9ac9be6640 1 librados: setting wanted keys 2024-03-29T11:30:17.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.895+0000 7f9ac9be6640 1 librados: calling monclient init 2024-03-29T11:30:17.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.895+0000 7f9ac9be6640 10 monclient: init 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac40a9750 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac4149ac0 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac414fe50 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac406f6f0 con 0x7f9ac40a9750 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac40720d0 con 0x7f9ac4149ac0 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac3fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f9ac414fe50 legacy=0x7f9ac41dec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac3fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f9ac414fe50 legacy=0x7f9ac41dec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:30:17.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac3fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9ac414fe50 0x7f9ac41dec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac3fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9ac414fe50 0x7f9ac41dec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac406f1d0 con 0x7f9ac414fe50 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.897+0000 7f9ac9be6640 10 monclient(hunting): authenticate will time out at 1548.037773s 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.900+0000 7f9ac0ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.52:6790/0 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.900+0000 7f9abbfff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f9ac4149ac0 0x7f9ac41403b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:41005/0 (socket says 172.21.15.52:41005) 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.900+0000 7f9abbfff640 1 -- 172.21.15.52:0/2893339984 learned_addr learned my addr 172.21.15.52:0/2893339984 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f9ac4149ac0 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196589590 0 0) 0x7f9ac40720d0 con 0x7f9ac4149ac0 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa4004eb0 con 0x7f9ac4149ac0 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f9ac40a9750 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989508728 0 0) 0x7f9ac40aa4a0 con 0x7f9ac40a9750 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac40720d0 con 0x7f9ac40a9750 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1332832436 0 0) 0x7f9aa4004eb0 con 0x7f9ac4149ac0 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac40aa4a0 con 0x7f9ac4149ac0 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9aac001b80 con 0x7f9ac4149ac0 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:17.899 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.901+0000 7f9ac0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209066590 0 0) 0x7f9ac40720d0 con 0x7f9ac40a9750 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa4004eb0 con 0x7f9ac40a9750 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9ab4001640 con 0x7f9ac40a9750 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1079140099 0 0) 0x7f9aa4004eb0 con 0x7f9ac40a9750 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 >> v1:172.21.15.52:6790/0 conn(0x7f9ac414fe50 legacy=0x7f9ac41dec80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 >> v1:172.21.15.79:6789/0 conn(0x7f9ac4149ac0 legacy=0x7f9ac41403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 1 monclient: found mon.a 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.902+0000 7f9ac0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac406f1d0 con 0x7f9ac40a9750 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac9be6640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac9be6640 10 monclient: _renew_subs 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac9be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac406eeb0 con 0x7f9ac40a9750 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac9be6640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.903+0000 7f9ac9be6640 10 monclient: _renew_subs 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.904+0000 7f9ac9be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.904+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac4146630 con 0x7f9ac40a9750 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.904+0000 7f9ac9be6640 1 librados: init done 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.906+0000 7f9ac9be6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.906+0000 7f9ac9be6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.906+0000 7f9ac9be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.906+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac40afe20 con 0x7f9ac40a9750 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ab40017a0 con 0x7f9ac40a9750 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f9ab4001bf0 con 0x7f9ac40a9750 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac0ff9640 10 monclient: dump: 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.907+0000 7f9ac27fc640 4 set_mon_vals no callback set 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.909+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f9ab4003610 con 0x7f9ac40a9750 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.910+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7f9ab4083390 con 0x7f9ac40a9750 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.910+0000 7f9ac0ff9640 10 client.4569.objecter ms_dispatch 0x7f9ac4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.910+0000 7f9ac0ff9640 3 client.4569.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.910+0000 7f9ac0ff9640 3 client.4569.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.910+0000 7f9ac0ff9640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.912+0000 7f9abbfff640 10 monclient: get_auth_request con 0x7f9aa406df70 auth_method 0 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.913+0000 7f9ac0ff9640 10 client.4569.objecter ms_handle_connect 0x7f9aa406df70 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.915+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ab404ec70 con 0x7f9ac40a9750 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.915+0000 7f9ac0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:17.915+0000 7f9ac0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:18.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.243+0000 7f9ac9be6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.244+0000 7f9ac9be6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.244+0000 7f9ac9be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:18.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.244+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac406f040 con 0x7f9ac40a9750 2024-03-29T11:30:18.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.245+0000 7f9ac0ff9640 1 -- 172.21.15.52:0/2893339984 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7f9ab4050ba0 con 0x7f9ac40a9750 2024-03-29T11:30:18.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.245+0000 7f9ac0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.245+0000 7f9ac0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:18.238 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:18.238 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:18.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.248+0000 7f9ac9be6640 10 librados: watch_flush enter 2024-03-29T11:30:18.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 10 librados: watch_flush exit 2024-03-29T11:30:18.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:30:18.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:30:18.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 >> v1:172.21.15.79:6816/407480584 conn(0x7f9aa406df70 legacy=0x7f9aa4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 10 monclient: shutdown 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 >> v1:172.21.15.52:6789/0 conn(0x7f9ac40a9750 legacy=0x7f9ac406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 shutdown_connections 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 >> 172.21.15.52:0/2893339984 conn(0x7f9ac40b35e0 msgr2=0x7f9ac4143af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 shutdown_connections 2024-03-29T11:30:18.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 -- 172.21.15.52:0/2893339984 wait complete. 2024-03-29T11:30:18.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:18.249+0000 7f9ac9be6640 1 librados: shutdown 2024-03-29T11:30:18.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:25.066 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.228+0000 7fb27e01f640 10 monclient: get_monmap_and_config 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.229+0000 7fb27e01f640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.229+0000 7fb27e01f640 10 monclient: monmap: 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:25.229936+0000 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:25.229936+0000 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:25.222 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:25.223 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:25.223 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:25.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.229+0000 7fb27e01f640 1 Processor -- start 2024-03-29T11:30:25.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.230+0000 7fb27e01f640 1 -- start start 2024-03-29T11:30:25.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.230+0000 7fb27e01f640 10 monclient: init 2024-03-29T11:30:25.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient(hunting): picked mon.noname-a con 0x7fb2700389e0 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient(hunting): picked mon.noname-c con 0x7fb270038dc0 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient(hunting): picked mon.noname-b con 0x7fb270030660 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2700c54c0 con 0x7fb2700389e0 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb270036430 con 0x7fb270038dc0 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb270039840 con 0x7fb270030660 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27e01f640 10 monclient(hunting): authenticate will time out at 1555.371368s 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27d01d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb2700389e0 0x7fb270033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33882/0 (socket says 172.21.15.52:33882) 2024-03-29T11:30:25.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.231+0000 7fb27d01d640 1 -- 172.21.15.52:0/937200893 learned_addr learned my addr 172.21.15.52:0/937200893 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:25.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688545901 0 0) 0x7fb270036430 con 0x7fb270038dc0 2024-03-29T11:30:25.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb264003520 con 0x7fb270038dc0 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317073081 0 0) 0x7fb270039840 con 0x7fb270030660 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.232+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb270036430 con 0x7fb270030660 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759385958 0 0) 0x7fb2700c54c0 con 0x7fb2700389e0 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb270039840 con 0x7fb2700389e0 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579089493 0 0) 0x7fb264003520 con 0x7fb270038dc0 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 >> v1:172.21.15.52:6790/0 conn(0x7fb270030660 legacy=0x7fb2700d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 >> v1:172.21.15.52:6789/0 conn(0x7fb2700389e0 legacy=0x7fb270033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 monclient: found mon.noname-c 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.79:6789/0 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270035690 con 0x7fb270038dc0 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb268001690 con 0x7fb270038dc0 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: dump: 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:25.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb27e01f640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb27e01f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.1 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2680017f0 con 0x7fb270038dc0 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb27e01f640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 1 -- 172.21.15.52:0/937200893 <== mon.1 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb268001c00 con 0x7fb270038dc0 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb26ffff640 10 monclient: dump: 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:25.231 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.233+0000 7fb27e01f640 4 set_mon_vals no callback set 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 10 monclient: shutdown 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 1 -- 172.21.15.52:0/937200893 >> v1:172.21.15.79:6789/0 conn(0x7fb270038dc0 legacy=0x7fb2700cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 1 -- 172.21.15.52:0/937200893 shutdown_connections 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 1 -- 172.21.15.52:0/937200893 >> 172.21.15.52:0/937200893 conn(0x7fb270005c00 msgr2=0x7fb270006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 1 -- 172.21.15.52:0/937200893 shutdown_connections 2024-03-29T11:30:25.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.234+0000 7fb27e01f640 1 -- 172.21.15.52:0/937200893 wait complete. 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.235+0000 7fb27e01f640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.235+0000 7fb27e01f640 10 monclient: monmap: 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:25.237 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 1 librados: starting msgr at 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 1 librados: starting objecter 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 1 Processor -- start 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 1 -- start start 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 1 librados: setting wanted keys 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 1 librados: calling monclient init 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.236+0000 7fb27e01f640 10 monclient: init 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient(hunting): picked mon.noname-b con 0x7fb270030660 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2700389e0 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:25.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient(hunting): picked mon.noname-a con 0x7fb27002d590 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2700bbc70 con 0x7fb27002d590 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27000b440 con 0x7fb270030660 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27000b6b0 con 0x7fb2700389e0 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27e01f640 10 monclient(hunting): authenticate will time out at 1555.377966s 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27d81e640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb27002d590 0x7fb27016c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:34504/0 (socket says 172.21.15.52:34504) 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.237+0000 7fb27d81e640 1 -- 172.21.15.52:0/1435451193 learned_addr learned my addr 172.21.15.52:0/1435451193 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 client.?.objecter ms_handle_connect 0x7fb270030660 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 client.?.objecter ms_handle_connect 0x7fb27002d590 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 client.?.objecter ms_handle_connect 0x7fb2700389e0 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100417261 0 0) 0x7fb27000b440 con 0x7fb270030660 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.238+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c004eb0 con 0x7fb270030660 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704986517 0 0) 0x7fb2700bbc70 con 0x7fb27002d590 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27000b440 con 0x7fb27002d590 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2223438081 0 0) 0x7fb25c004eb0 con 0x7fb270030660 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2700bbc70 con 0x7fb270030660 2024-03-29T11:30:25.240 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb274001850 con 0x7fb270030660 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): dump: 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:25.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2059835692 0 0) 0x7fb2700bbc70 con 0x7fb270030660 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 >> v1:172.21.15.52:6790/0 conn(0x7fb2700389e0 legacy=0x7fb27000c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 >> v1:172.21.15.52:6789/0 conn(0x7fb27002d590 legacy=0x7fb27016c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.239+0000 7fb26dffb640 1 monclient: found mon.b 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27000b990 con 0x7fb270030660 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 10 monclient: _renew_subs 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:25.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27000d490 con 0x7fb270030660 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 10 monclient: _renew_subs 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27016d750 con 0x7fb270030660 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb27e01f640 1 librados: init done 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2740019b0 con 0x7fb270030660 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.240+0000 7fb26f7fe640 4 set_mon_vals no callback set 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7fb2740049c0 con 0x7fb270030660 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb26dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb26dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb26dffb640 10 monclient: dump: 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:25.243 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb27e01f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb27e01f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb27e01f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.241+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb258005080 con 0x7fb270030660 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.242+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7fb27401aa10 con 0x7fb270030660 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7fb274084a30 con 0x7fb270030660 2024-03-29T11:30:25.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb26dffb640 10 client.4552.objecter ms_dispatch 0x7fb278000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb26dffb640 3 client.4552.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb26dffb640 3 client.4552.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb26dffb640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb27c81c640 10 monclient: get_auth_request con 0x7fb25c06dd40 auth_method 0 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.243+0000 7fb26dffb640 10 client.4552.objecter ms_handle_connect 0x7fb25c06dd40 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.249+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb274050140 con 0x7fb270030660 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.249+0000 7fb26dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.249+0000 7fb26dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:25.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.630+0000 7fb27e01f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.630+0000 7fb27e01f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.630+0000 7fb27e01f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:25.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.630+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb258005330 con 0x7fb270030660 2024-03-29T11:30:25.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.631+0000 7fb26dffb640 1 -- 172.21.15.52:0/1435451193 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7fb274052240 con 0x7fb270030660 2024-03-29T11:30:25.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.631+0000 7fb26dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.631+0000 7fb26dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:25.625 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:25.625 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:25.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.635+0000 7fb27e01f640 10 librados: watch_flush enter 2024-03-29T11:30:25.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.635+0000 7fb27e01f640 10 librados: watch_flush exit 2024-03-29T11:30:25.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.635+0000 7fb27e01f640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:30:25.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.635+0000 7fb27e01f640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.635+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 >> v1:172.21.15.79:6816/407480584 conn(0x7fb25c06dd40 legacy=0x7fb25c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.635+0000 7fb27e01f640 10 monclient: shutdown 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 >> v1:172.21.15.79:6789/0 conn(0x7fb270030660 legacy=0x7fb27000c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 shutdown_connections 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 >> 172.21.15.52:0/1435451193 conn(0x7fb270005c00 msgr2=0x7fb2700ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 shutdown_connections 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 1 -- 172.21.15.52:0/1435451193 wait complete. 2024-03-29T11:30:25.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:25.636+0000 7fb27e01f640 1 librados: shutdown 2024-03-29T11:30:25.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:32.453 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:32.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.633+0000 7f7a6d9e6640 10 monclient: get_monmap_and_config 2024-03-29T11:30:32.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.633+0000 7f7a6d9e6640 10 monclient: build_initial_monmap 2024-03-29T11:30:32.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.633+0000 7f7a6d9e6640 10 monclient: monmap: 2024-03-29T11:30:32.626 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:32.626 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:32.634452+0000 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:32.634452+0000 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.634+0000 7f7a6d9e6640 1 Processor -- start 2024-03-29T11:30:32.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.634+0000 7f7a6d9e6640 1 -- start start 2024-03-29T11:30:32.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.634+0000 7f7a6d9e6640 10 monclient: init 2024-03-29T11:30:32.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:32.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:32.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7a681457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f7a6813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f7a680a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680aaef0 con 0x7f7a681457f0 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a68145bd0 con 0x7f7a680a9d70 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a68145eb0 con 0x7f7a6813f650 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.635+0000 7f7a6d9e6640 10 monclient(hunting): authenticate will time out at 1562.775858s 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a667fc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f7a6813f650 0x7f7a680b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56302/0 (socket says 172.21.15.52:56302) 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a667fc640 1 -- 172.21.15.52:0/1499190280 learned_addr learned my addr 172.21.15.52:0/1499190280 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:32.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231920629 0 0) 0x7f7a68145bd0 con 0x7f7a680a9d70 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50003520 con 0x7f7a680a9d70 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935627317 0 0) 0x7f7a68145eb0 con 0x7f7a6813f650 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:30:32.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.636+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a68145bd0 con 0x7f7a6813f650 2024-03-29T11:30:32.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526420492 0 0) 0x7f7a680aaef0 con 0x7f7a681457f0 2024-03-29T11:30:32.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a68145eb0 con 0x7f7a681457f0 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127722694 0 0) 0x7f7a68145bd0 con 0x7f7a6813f650 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 >> v1:172.21.15.79:6789/0 conn(0x7f7a680a9d70 legacy=0x7f7a6814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 >> v1:172.21.15.52:6789/0 conn(0x7f7a681457f0 legacy=0x7f7a6813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 monclient: found mon.noname-b 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a68146340 con 0x7f7a6813f650 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7a5c001670 con 0x7f7a6813f650 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: dump: 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a6d9e6640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a6d9e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a5c0017d0 con 0x7f7a6813f650 2024-03-29T11:30:32.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a6d9e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.637+0000 7f7a6d9e6640 4 set_mon_vals no callback set 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a65ffb640 1 -- 172.21.15.52:0/1499190280 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7a5c001c20 con 0x7f7a6813f650 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a65ffb640 10 monclient: dump: 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:32.634 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a6d9e6640 10 monclient: shutdown 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a6d9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/1499190280 >> v1:172.21.15.52:6790/0 conn(0x7f7a6813f650 legacy=0x7f7a680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.638+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/1499190280 shutdown_connections 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.639+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/1499190280 >> 172.21.15.52:0/1499190280 conn(0x7f7a68066a00 msgr2=0x7f7a68066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.639+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/1499190280 shutdown_connections 2024-03-29T11:30:32.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.639+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/1499190280 wait complete. 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.639+0000 7f7a6d9e6640 10 monclient: build_initial_monmap 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.639+0000 7f7a6d9e6640 10 monclient: monmap: 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:32.638 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 1 librados: starting msgr at 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 1 librados: starting objecter 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 1 Processor -- start 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 1 -- start start 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 1 librados: setting wanted keys 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 1 librados: calling monclient init 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.640+0000 7f7a6d9e6640 10 monclient: init 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.641+0000 7f7a6d9e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.641+0000 7f7a6d9e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:32.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.641+0000 7f7a6d9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f7a680a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.641+0000 7f7a66ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f7a680a9d70 0x7f7a681d71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.52:55332/0 (socket says 172.21.15.52:55332) 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.641+0000 7f7a66ffd640 1 -- 172.21.15.52:0/2946422998 learned_addr learned my addr 172.21.15.52:0/2946422998 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.641+0000 7f7a6d9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f7a6813f650 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a6d9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7a681457f0 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 client.?.objecter ms_handle_connect 0x7f7a680a9d70 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a681d4e30 con 0x7f7a681457f0 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a680b0070 con 0x7f7a680a9d70 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a681d7a50 con 0x7f7a6813f650 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a6d9e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a6d9e6640 10 monclient(hunting): authenticate will time out at 1562.782363s 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 client.?.objecter ms_handle_connect 0x7f7a681457f0 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905184944 0 0) 0x7f7a680b0070 con 0x7f7a680a9d70 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a54004eb0 con 0x7f7a680a9d70 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 client.?.objecter ms_handle_connect 0x7f7a6813f650 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2263355118 0 0) 0x7f7a54004eb0 con 0x7f7a680a9d70 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a680b0070 con 0x7f7a680a9d70 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7a58001970 con 0x7f7a680a9d70 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): dump: 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:32.641 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375539472 0 0) 0x7f7a681d4e30 con 0x7f7a681457f0 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.642+0000 7f7a4ffff640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a54004eb0 con 0x7f7a681457f0 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368285550 0 0) 0x7f7a681d7a50 con 0x7f7a6813f650 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a681d4e30 con 0x7f7a6813f650 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380543540 0 0) 0x7f7a54004eb0 con 0x7f7a681457f0 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a681d7a50 con 0x7f7a681457f0 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7a60001c50 con 0x7f7a681457f0 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient(hunting): dump: 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:32.643 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1616555989 0 0) 0x7f7a680b0070 con 0x7f7a680a9d70 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 >> v1:172.21.15.52:6790/0 conn(0x7f7a6813f650 legacy=0x7f7a680adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 >> v1:172.21.15.52:6789/0 conn(0x7f7a681457f0 legacy=0x7f7a681e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 monclient: found mon.b 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:32.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680b1490 con 0x7f7a680a9d70 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 10 monclient: _renew_subs 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a681d7f70 con 0x7f7a680a9d70 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 10 monclient: _renew_subs 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a680b0c50 con 0x7f7a680a9d70 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.643+0000 7f7a6d9e6640 1 librados: init done 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a58003610 con 0x7f7a680a9d70 2024-03-29T11:30:32.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f7a580039e0 con 0x7f7a680a9d70 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a657fa640 4 set_mon_vals no callback set 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.644+0000 7f7a4ffff640 10 monclient: dump: 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:32.646 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:32.647 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.646+0000 7f7a6d9e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.646+0000 7f7a6d9e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.647+0000 7f7a6d9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.647+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a48005080 con 0x7f7a680a9d70 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.647+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f7a58003b40 con 0x7f7a680a9d70 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7f7a58084a10 con 0x7f7a680a9d70 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a4ffff640 10 client.4558.objecter ms_dispatch 0x7f7a68000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a4ffff640 3 client.4558.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:32.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a4ffff640 3 client.4558.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:32.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a4ffff640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-29T11:30:32.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a667fc640 10 monclient: get_auth_request con 0x7f7a54076860 auth_method 0 2024-03-29T11:30:32.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.648+0000 7f7a4ffff640 10 client.4558.objecter ms_handle_connect 0x7f7a54076860 2024-03-29T11:30:32.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.654+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a580502f0 con 0x7f7a680a9d70 2024-03-29T11:30:32.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.654+0000 7f7a4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:32.654+0000 7f7a4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:33.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.077+0000 7f7a6d9e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.077+0000 7f7a6d9e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.077+0000 7f7a6d9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.79:6789/0 2024-03-29T11:30:33.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.077+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a48005330 con 0x7f7a680a9d70 2024-03-29T11:30:33.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.078+0000 7f7a4ffff640 1 -- 172.21.15.52:0/2946422998 <== mon.1 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7f7a58052220 con 0x7f7a680a9d70 2024-03-29T11:30:33.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.078+0000 7f7a4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.078+0000 7f7a4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:33.071 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:33.071 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:33.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 10 librados: watch_flush enter 2024-03-29T11:30:33.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 10 librados: watch_flush exit 2024-03-29T11:30:33.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-29T11:30:33.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 10 client.4558.objecter successfully canceled tick 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 >> v1:172.21.15.79:6816/407480584 conn(0x7f7a54076860 legacy=0x7f7a54078d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 10 monclient: shutdown 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 >> v1:172.21.15.79:6789/0 conn(0x7f7a680a9d70 legacy=0x7f7a681d71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 shutdown_connections 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.083+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 >> 172.21.15.52:0/2946422998 conn(0x7f7a68066a00 msgr2=0x7f7a6813c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.084+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 shutdown_connections 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.084+0000 7f7a6d9e6640 1 -- 172.21.15.52:0/2946422998 wait complete. 2024-03-29T11:30:33.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:33.084+0000 7f7a6d9e6640 1 librados: shutdown 2024-03-29T11:30:33.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:39.898 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:40.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.055+0000 7f2aa28d6640 10 monclient: get_monmap_and_config 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.055+0000 7f2aa28d6640 10 monclient: build_initial_monmap 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.055+0000 7f2aa28d6640 10 monclient: monmap: 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:40.056858+0000 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:40.056858+0000 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:40.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.057+0000 7f2aa28d6640 1 Processor -- start 2024-03-29T11:30:40.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.057+0000 7f2aa28d6640 1 -- start start 2024-03-29T11:30:40.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.057+0000 7f2aa28d6640 10 monclient: init 2024-03-29T11:30:40.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:40.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:40.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c0ae250 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:40.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9c13f970 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:40.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c142800 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:40.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0afe00 con 0x7f2a9c0ae250 2024-03-29T11:30:40.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c142c10 con 0x7f2a9c142800 2024-03-29T11:30:40.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0afb20 con 0x7f2a9c13f970 2024-03-29T11:30:40.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:40.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.058+0000 7f2aa28d6640 10 monclient(hunting): authenticate will time out at 1570.198786s 2024-03-29T11:30:40.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.059+0000 7f2a9bfff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f2a9c0ae250 0x7f2a9c13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:48236/0 (socket says 172.21.15.52:48236) 2024-03-29T11:30:40.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.059+0000 7f2a9bfff640 1 -- 172.21.15.52:0/1903613409 learned_addr learned my addr 172.21.15.52:0/1903613409 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.060+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970336400 0 0) 0x7f2a9c0afe00 con 0x7f2a9c0ae250 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.060+0000 7f2a9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.060+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.060+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.060+0000 7f2a9affd640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.060+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a88003520 con 0x7f2a9c0ae250 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.061+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2107430972 0 0) 0x7f2a88003520 con 0x7f2a9c0ae250 2024-03-29T11:30:40.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.061+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 >> v1:172.21.15.52:6790/0 conn(0x7f2a9c13f970 legacy=0x7f2a9c1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:40.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 >> v1:172.21.15.79:6789/0 conn(0x7f2a9c142800 legacy=0x7f2a9c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 1 monclient: found mon.noname-a 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0aab80 con 0x7f2a9c0ae250 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: _finish_auth 0 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2a94001a10 con 0x7f2a9c0ae250 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.062+0000 7f2a9affd640 10 monclient: dump: 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:40.057 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.063+0000 7f2aa28d6640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.063+0000 7f2aa28d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.063+0000 7f2a9affd640 1 -- 172.21.15.52:0/1903613409 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a94001ca0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.064+0000 7f2a9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:40.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.064+0000 7f2aa28d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:40.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.064+0000 7f2aa28d6640 4 set_mon_vals no callback set 2024-03-29T11:30:40.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.064+0000 7f2aa28d6640 10 monclient: shutdown 2024-03-29T11:30:40.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.064+0000 7f2aa28d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.064+0000 7f2aa28d6640 1 -- 172.21.15.52:0/1903613409 >> v1:172.21.15.52:6789/0 conn(0x7f2a9c0ae250 legacy=0x7f2a9c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.065+0000 7f2aa28d6640 1 -- 172.21.15.52:0/1903613409 shutdown_connections 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.065+0000 7f2aa28d6640 1 -- 172.21.15.52:0/1903613409 >> 172.21.15.52:0/1903613409 conn(0x7f2a9c0b35e0 msgr2=0x7f2a9c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.065+0000 7f2aa28d6640 1 -- 172.21.15.52:0/1903613409 shutdown_connections 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.066+0000 7f2aa28d6640 1 -- 172.21.15.52:0/1903613409 wait complete. 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 10 monclient: build_initial_monmap 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 10 monclient: monmap: 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:40.060 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 1 librados: starting msgr at 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 1 librados: starting objecter 2024-03-29T11:30:40.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 1 Processor -- start 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 1 -- start start 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 1 librados: setting wanted keys 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 1 librados: calling monclient init 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.067+0000 7f2aa28d6640 10 monclient: init 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c0ae250 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c13f970 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9c142800 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c1434f0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c143760 con 0x7f2a9c142800 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c1e4330 con 0x7f2a9c13f970 2024-03-29T11:30:40.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:40.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.068+0000 7f2aa28d6640 10 monclient(hunting): authenticate will time out at 1570.208857s 2024-03-29T11:30:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.069+0000 7f2a9bfff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f2a9c0ae250 0x7f2a9c1e1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:50772/0 (socket says 172.21.15.52:50772) 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.069+0000 7f2a9bfff640 1 -- 172.21.15.52:0/3621038272 learned_addr learned my addr 172.21.15.52:0/3621038272 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.069+0000 7f2a98ff9640 10 client.?.objecter ms_handle_connect 0x7f2a9c0ae250 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.069+0000 7f2a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880844503 0 0) 0x7f2a9c1434f0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c004eb0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.072+0000 7f2a98ff9640 10 client.?.objecter ms_handle_connect 0x7f2a9c142800 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086134284 0 0) 0x7f2a9c143760 con 0x7f2a9c142800 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a9c1434f0 con 0x7f2a9c142800 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 client.?.objecter ms_handle_connect 0x7f2a9c13f970 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338663617 0 0) 0x7f2a7c004eb0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a9c143760 con 0x7f2a9c0ae250 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2a94004cc0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:40.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372580723 0 0) 0x7f2a9c1e4330 con 0x7f2a9c13f970 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c004eb0 con 0x7f2a9c13f970 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 940768547 0 0) 0x7f2a9c143760 con 0x7f2a9c0ae250 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 >> v1:172.21.15.52:6790/0 conn(0x7f2a9c13f970 legacy=0x7f2a9c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 >> v1:172.21.15.79:6789/0 conn(0x7f2a9c142800 legacy=0x7f2a9c1ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 monclient: found mon.a 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.073+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c146440 con 0x7f2a9c0ae250 2024-03-29T11:30:40.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 10 monclient: _renew_subs 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c1e3da0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 10 monclient: _renew_subs 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c138fb0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2aa28d6640 1 librados: init done 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a940050d0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f2a94002030 con 0x7f2a9c0ae250 2024-03-29T11:30:40.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.074+0000 7f2a98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.075+0000 7f2a98ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.075+0000 7f2a98ff9640 10 monclient: dump: 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.075+0000 7f2a9a7fc640 4 set_mon_vals no callback set 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2aa28d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f2a94005240 con 0x7f2a9c0ae250 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7f2a94084fa0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2a98ff9640 10 client.4587.objecter ms_dispatch 0x7f2a9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2a98ff9640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2a98ff9640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.076+0000 7f2a98ff9640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.077+0000 7f2aa28d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.077+0000 7f2aa28d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.077+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a80005080 con 0x7f2a9c0ae250 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.077+0000 7f2a9b7fe640 10 monclient: get_auth_request con 0x7f2a7c06dc60 auth_method 0 2024-03-29T11:30:40.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.078+0000 7f2a98ff9640 10 client.4587.objecter ms_handle_connect 0x7f2a7c06dc60 2024-03-29T11:30:40.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.085+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a94050880 con 0x7f2a9c0ae250 2024-03-29T11:30:40.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.085+0000 7f2a98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.085+0000 7f2a98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:40.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.423+0000 7f2aa28d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.423+0000 7f2aa28d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.423+0000 7f2aa28d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:40.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.423+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a80005330 con 0x7f2a9c0ae250 2024-03-29T11:30:40.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.424+0000 7f2a98ff9640 1 -- 172.21.15.52:0/3621038272 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7f2a940527b0 con 0x7f2a9c0ae250 2024-03-29T11:30:40.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.424+0000 7f2a98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.424+0000 7f2a98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:40.420 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:40.420 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 10 librados: watch_flush enter 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 10 librados: watch_flush exit 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 >> v1:172.21.15.79:6816/407480584 conn(0x7f2a7c06dc60 legacy=0x7f2a7c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 10 monclient: shutdown 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 >> v1:172.21.15.52:6789/0 conn(0x7f2a9c0ae250 legacy=0x7f2a9c1e1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 shutdown_connections 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 >> 172.21.15.52:0/3621038272 conn(0x7f2a9c0b35e0 msgr2=0x7f2a9c0b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.430+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 shutdown_connections 2024-03-29T11:30:40.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.431+0000 7f2aa28d6640 1 -- 172.21.15.52:0/3621038272 wait complete. 2024-03-29T11:30:40.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:40.431+0000 7f2aa28d6640 1 librados: shutdown 2024-03-29T11:30:40.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:47.249 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.405+0000 7faf62947640 10 monclient: get_monmap_and_config 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.405+0000 7faf62947640 10 monclient: build_initial_monmap 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.405+0000 7faf62947640 10 monclient: monmap: 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:47.406586+0000 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:47.406586+0000 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:47.399 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.406+0000 7faf62947640 1 Processor -- start 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.406+0000 7faf62947640 1 -- start start 2024-03-29T11:30:47.400 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.406+0000 7faf62947640 10 monclient: init 2024-03-29T11:30:47.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:47.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:47.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient(hunting): picked mon.noname-c con 0x7faf5c06d9d0 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient(hunting): picked mon.noname-b con 0x7faf5c13e820 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient(hunting): picked mon.noname-a con 0x7faf5c0af1d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c0aed10 con 0x7faf5c0af1d0 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c0afd50 con 0x7faf5c06d9d0 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c0ab490 con 0x7faf5c13e820 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.407+0000 7faf62947640 10 monclient(hunting): authenticate will time out at 1577.547896s 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf60ee4640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7faf5c0af1d0 0x7faf5c0af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46734/0 (socket says 172.21.15.52:46734) 2024-03-29T11:30:47.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf60ee4640 1 -- 172.21.15.52:0/2869205798 learned_addr learned my addr 172.21.15.52:0/2869205798 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727543037 0 0) 0x7faf5c0aed10 con 0x7faf5c0af1d0 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf5affd640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.408+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4400bd30 con 0x7faf5c0af1d0 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.409+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560978446 0 0) 0x7faf4400bd30 con 0x7faf5c0af1d0 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.409+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 >> v1:172.21.15.52:6790/0 conn(0x7faf5c13e820 legacy=0x7faf5c13ec50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 >> v1:172.21.15.79:6789/0 conn(0x7faf5c06d9d0 legacy=0x7faf5c0b05f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 1 monclient: found mon.noname-a 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0ab770 con 0x7faf5c0af1d0 2024-03-29T11:30:47.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: _finish_auth 0 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf62947640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf62947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf54001a10 con 0x7faf5c0af1d0 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: dump: 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:47.408 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf62947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 1 -- 172.21.15.52:0/2869205798 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf54001ca0 con 0x7faf5c0af1d0 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf62947640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.412+0000 7faf62947640 4 set_mon_vals no callback set 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 10 monclient: shutdown 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 1 -- 172.21.15.52:0/2869205798 >> v1:172.21.15.52:6789/0 conn(0x7faf5c0af1d0 legacy=0x7faf5c0af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 1 -- 172.21.15.52:0/2869205798 shutdown_connections 2024-03-29T11:30:47.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 1 -- 172.21.15.52:0/2869205798 >> 172.21.15.52:0/2869205798 conn(0x7faf5c06b660 msgr2=0x7faf5c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 1 -- 172.21.15.52:0/2869205798 shutdown_connections 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.413+0000 7faf62947640 1 -- 172.21.15.52:0/2869205798 wait complete. 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.414+0000 7faf62947640 10 monclient: build_initial_monmap 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.414+0000 7faf62947640 10 monclient: monmap: 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:47.412 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 1 librados: starting msgr at 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 1 librados: starting objecter 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 1 Processor -- start 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 1 -- start start 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 1 librados: setting wanted keys 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 1 librados: calling monclient init 2024-03-29T11:30:47.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.415+0000 7faf62947640 10 monclient: init 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.416+0000 7faf62947640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.417+0000 7faf62947640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.417+0000 7faf62947640 10 monclient(hunting): picked mon.noname-a con 0x7faf5c06d9d0 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.417+0000 7faf62947640 10 monclient(hunting): picked mon.noname-c con 0x7faf5c0af1d0 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.418+0000 7faf62947640 10 monclient(hunting): picked mon.noname-b con 0x7faf5c13e820 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.418+0000 7faf62947640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c137780 con 0x7faf5c06d9d0 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.418+0000 7faf62947640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c137a60 con 0x7faf5c13e820 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.418+0000 7faf62947640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5c137d40 con 0x7faf5c0af1d0 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.418+0000 7faf62947640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:47.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.418+0000 7faf62947640 10 monclient(hunting): authenticate will time out at 1577.558223s 2024-03-29T11:30:47.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf5b7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7faf5c0af1d0 0x7faf5c136990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39439/0 (socket says 172.21.15.52:39439) 2024-03-29T11:30:47.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf5b7fe640 1 -- 172.21.15.52:0/4001133363 learned_addr learned my addr 172.21.15.52:0/4001133363 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:47.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf5bfff640 1 --1- 172.21.15.52:0/4001133363 >> v1:172.21.15.52:6789/0 conn(0x7faf5c06d9d0 0x7faf5c136280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:47304/0 (socket says 172.21.15.52:47304) 2024-03-29T11:30:47.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf58ff9640 10 client.?.objecter ms_handle_connect 0x7faf5c0af1d0 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf58ff9640 10 client.?.objecter ms_handle_connect 0x7faf5c06d9d0 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.420+0000 7faf58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520614209 0 0) 0x7faf5c137d40 con 0x7faf5c0af1d0 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf48004eb0 con 0x7faf5c0af1d0 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 client.?.objecter ms_handle_connect 0x7faf5c13e820 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956863896 0 0) 0x7faf5c137780 con 0x7faf5c06d9d0 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.421+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c137d40 con 0x7faf5c06d9d0 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259890325 0 0) 0x7faf5c137d40 con 0x7faf5c06d9d0 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf5c137780 con 0x7faf5c06d9d0 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf4c001750 con 0x7faf5c06d9d0 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:47.422 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:47.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1998048804 0 0) 0x7faf5c137780 con 0x7faf5c06d9d0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 >> v1:172.21.15.52:6790/0 conn(0x7faf5c0af1d0 legacy=0x7faf5c136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 >> v1:172.21.15.79:6789/0 conn(0x7faf5c13e820 legacy=0x7faf5c1eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 monclient: found mon.a 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0b2fc0 con 0x7faf5c06d9d0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.423+0000 7faf58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf4c0018b0 con 0x7faf5c06d9d0 2024-03-29T11:30:47.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7faf4c004940 con 0x7faf5c06d9d0 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf58ff9640 10 monclient: dump: 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:47.425 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 10 monclient: _renew_subs 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c0b34e0 con 0x7faf5c06d9d0 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf59ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 10 monclient: _renew_subs 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c138520 con 0x7faf5c06d9d0 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.424+0000 7faf62947640 1 librados: init done 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.426+0000 7faf62947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.426+0000 7faf62947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.426+0000 7faf62947640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:47.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.426+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c075f10 con 0x7faf5c06d9d0 2024-03-29T11:30:47.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.427+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7faf4c01a910 con 0x7faf5c06d9d0 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7faf4c084860 con 0x7faf5c06d9d0 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf5b7fe640 10 monclient: get_auth_request con 0x7faf4806dde0 auth_method 0 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf58ff9640 10 client.4593.objecter ms_dispatch 0x7faf5c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf58ff9640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf58ff9640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf58ff9640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.428+0000 7faf58ff9640 10 client.4593.objecter ms_handle_connect 0x7faf4806dde0 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.434+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf4c050140 con 0x7faf5c06d9d0 2024-03-29T11:30:47.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.434+0000 7faf58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.434+0000 7faf58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:47.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.770+0000 7faf62947640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.770+0000 7faf62947640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.770+0000 7faf62947640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-29T11:30:47.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.770+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf5c0b3150 con 0x7faf5c06d9d0 2024-03-29T11:30:47.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.771+0000 7faf58ff9640 1 -- 172.21.15.52:0/4001133363 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7faf4c052070 con 0x7faf5c06d9d0 2024-03-29T11:30:47.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.771+0000 7faf58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.771+0000 7faf58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:47.765 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:47.765 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.774+0000 7faf62947640 10 librados: watch_flush enter 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.774+0000 7faf62947640 10 librados: watch_flush exit 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.774+0000 7faf62947640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.774+0000 7faf62947640 10 client.4593.objecter successfully canceled tick 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.775+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 >> v1:172.21.15.79:6816/407480584 conn(0x7faf4806dde0 legacy=0x7faf480702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.775+0000 7faf62947640 10 monclient: shutdown 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.775+0000 7faf62947640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.775+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 >> v1:172.21.15.52:6789/0 conn(0x7faf5c06d9d0 legacy=0x7faf5c136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.775+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 shutdown_connections 2024-03-29T11:30:47.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.775+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 >> 172.21.15.52:0/4001133363 conn(0x7faf5c06b660 msgr2=0x7faf5c13ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:47.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.777+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 shutdown_connections 2024-03-29T11:30:47.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.777+0000 7faf62947640 1 -- 172.21.15.52:0/4001133363 wait complete. 2024-03-29T11:30:47.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:47.777+0000 7faf62947640 1 librados: shutdown 2024-03-29T11:30:47.790 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:54.597 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:54.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.753+0000 7f1dbd7ef640 10 monclient: get_monmap_and_config 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.753+0000 7f1dbd7ef640 10 monclient: build_initial_monmap 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.753+0000 7f1dbd7ef640 10 monclient: monmap: 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:30:54.754080+0000 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:30:54.754080+0000 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:54.747 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-c 2024-03-29T11:30:54.748 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-29T11:30:54.748 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.754+0000 7f1dbd7ef640 1 Processor -- start 2024-03-29T11:30:54.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.754+0000 7f1dbd7ef640 1 -- start start 2024-03-29T11:30:54.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.754+0000 7f1dbd7ef640 10 monclient: init 2024-03-29T11:30:54.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:54.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:54.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient(hunting): picked mon.noname-c con 0x7f1db81457f0 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:54.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient(hunting): picked mon.noname-a con 0x7f1db813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:54.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient(hunting): picked mon.noname-b con 0x7f1db80a9d70 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:54.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db80aaef0 con 0x7f1db813f650 2024-03-29T11:30:54.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db8145bd0 con 0x7f1db81457f0 2024-03-29T11:30:54.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db8145eb0 con 0x7f1db80a9d70 2024-03-29T11:30:54.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:54.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.755+0000 7f1dbd7ef640 10 monclient(hunting): authenticate will time out at 1584.895844s 2024-03-29T11:30:54.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.756+0000 7f1db67fc640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f1db813f650 0x7f1db80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:52482/0 (socket says 172.21.15.52:52482) 2024-03-29T11:30:54.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.756+0000 7f1db67fc640 1 -- 172.21.15.52:0/1297225618 learned_addr learned my addr 172.21.15.52:0/1297225618 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:54.753 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.758+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94162465 0 0) 0x7f1db80aaef0 con 0x7f1db813f650 2024-03-29T11:30:54.753 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.758+0000 7f1db5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.758+0000 7f1db5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.758+0000 7f1db5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.758+0000 7f1db5ffb640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.758+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da0003520 con 0x7f1db813f650 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241305338 0 0) 0x7f1da0003520 con 0x7f1db813f650 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 >> v1:172.21.15.52:6790/0 conn(0x7f1db80a9d70 legacy=0x7f1db814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 >> v1:172.21.15.79:6789/0 conn(0x7f1db81457f0 legacy=0x7f1db813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:30:54.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db8146340 con 0x7f1db813f650 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1da8001690 con 0x7f1db813f650 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1db5ffb640 10 monclient: dump: 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.755 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1dbd7ef640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:30:54.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.759+0000 7f1dbd7ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:54.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1da80017f0 con 0x7f1db813f650 2024-03-29T11:30:54.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1db5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:54.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1db5ffb640 1 -- 172.21.15.52:0/1297225618 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1da8001c40 con 0x7f1db813f650 2024-03-29T11:30:54.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1db5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1db5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1db5ffb640 10 monclient: dump: 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1dbd7ef640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.761+0000 7f1dbd7ef640 4 set_mon_vals no callback set 2024-03-29T11:30:54.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 10 monclient: shutdown 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1297225618 >> v1:172.21.15.52:6789/0 conn(0x7f1db813f650 legacy=0x7f1db80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1297225618 shutdown_connections 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1297225618 >> 172.21.15.52:0/1297225618 conn(0x7f1db8066a00 msgr2=0x7f1db8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1297225618 shutdown_connections 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.762+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1297225618 wait complete. 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.763+0000 7f1dbd7ef640 10 monclient: build_initial_monmap 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.763+0000 7f1dbd7ef640 10 monclient: monmap: 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-29T11:30:54.760 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.noname-b 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.763+0000 7f1dbd7ef640 1 librados: starting msgr at 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.763+0000 7f1dbd7ef640 1 librados: starting objecter 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.764+0000 7f1dbd7ef640 1 Processor -- start 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.764+0000 7f1dbd7ef640 1 -- start start 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.764+0000 7f1dbd7ef640 1 librados: setting wanted keys 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.764+0000 7f1dbd7ef640 1 librados: calling monclient init 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.764+0000 7f1dbd7ef640 10 monclient: init 2024-03-29T11:30:54.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.765+0000 7f1dbd7ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:54.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.765+0000 7f1dbd7ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:54.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 10 monclient(hunting): picked mon.noname-b con 0x7f1db80a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T11:30:54.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 10 monclient(hunting): picked mon.noname-a con 0x7f1db813f650 addr v1:172.21.15.52:6789/0 2024-03-29T11:30:54.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 10 monclient(hunting): picked mon.noname-c con 0x7f1db81457f0 addr v1:172.21.15.52:6790/0 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db81dc5d0 con 0x7f1db813f650 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db806f010 con 0x7f1db80a9d70 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db806f2f0 con 0x7f1db81457f0 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1dbd7ef640 10 monclient(hunting): authenticate will time out at 1584.907528s 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1db67fc640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f1db813f650 0x7f1db8137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:52880/0 (socket says 172.21.15.52:52880) 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1db67fc640 1 -- 172.21.15.52:0/1582443233 learned_addr learned my addr 172.21.15.52:0/1582443233 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1d9f7fe640 10 client.?.objecter ms_handle_connect 0x7f1db813f650 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.767+0000 7f1d9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.768+0000 7f1d9f7fe640 10 client.?.objecter ms_handle_connect 0x7f1db80a9d70 2024-03-29T11:30:54.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.768+0000 7f1d9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.768+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057415143 0 0) 0x7f1db81dc5d0 con 0x7f1db813f650 2024-03-29T11:30:54.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da4004eb0 con 0x7f1db813f650 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 10 client.?.objecter ms_handle_connect 0x7f1db81457f0 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.770+0000 7f1d9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.1 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977068786 0 0) 0x7f1db806f010 con 0x7f1db80a9d70 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db81dc5d0 con 0x7f1db80a9d70 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984202914 0 0) 0x7f1db806f2f0 con 0x7f1db81457f0 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db806f010 con 0x7f1db81457f0 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000970922 0 0) 0x7f1da4004eb0 con 0x7f1db813f650 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1da4005620 con 0x7f1db813f650 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1da8001b00 con 0x7f1db813f650 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.772 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4036664377 0 0) 0x7f1db806f010 con 0x7f1db81457f0 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1da4004eb0 con 0x7f1db81457f0 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1dac001770 con 0x7f1db81457f0 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.771+0000 7f1d9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:54.773 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1802313420 0 0) 0x7f1da4004eb0 con 0x7f1db81457f0 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 >> v1:172.21.15.79:6789/0 conn(0x7f1db80a9d70 legacy=0x7f1db81dbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 >> v1:172.21.15.52:6789/0 conn(0x7f1db813f650 legacy=0x7f1db8137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:54.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 1 monclient: found mon.c 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db81db6c0 con 0x7f1db81457f0 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1d9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 10 monclient: _renew_subs 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db81db9a0 con 0x7f1db81457f0 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 10 monclient: _renew_subs 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:54.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db8142d90 con 0x7f1db81457f0 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.772+0000 7f1dbd7ef640 1 librados: init done 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dac001c20 con 0x7f1db81457f0 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1d9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3468229668 0 0) 0x7f1dac003790 con 0x7f1db81457f0 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1d9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1d9f7fe640 10 monclient: dump: 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:fsid 3b9c51c5-3002-4ea5-be53-d626ccc5bde4 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-29T11:19:20.814219+0000 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-29T11:30:54.776 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.79:6789/0 mon.b 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1db57fa640 4 set_mon_vals no callback set 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.773+0000 7f1dbd7ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.774+0000 7f1dbd7ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.775+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2250616729 0 0) 0x7f1dac019760 con 0x7f1db81457f0 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1dbd7ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d98005080 con 0x7f1db81457f0 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1653161919 0 0) 0x7f1dac0835c0 con 0x7f1db81457f0 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1d9f7fe640 10 client.4670.objecter ms_dispatch 0x7f1db8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:54.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1d9f7fe640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:54.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1d9f7fe640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:54.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.776+0000 7f1d9f7fe640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T11:30:54.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.778+0000 7f1db6ffd640 10 monclient: get_auth_request con 0x7f1da406e130 auth_method 0 2024-03-29T11:30:54.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.780+0000 7f1d9f7fe640 10 client.4670.objecter ms_handle_connect 0x7f1da406e130 2024-03-29T11:30:54.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.784+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dac04ef00 con 0x7f1db81457f0 2024-03-29T11:30:54.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.784+0000 7f1d9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:54.784+0000 7f1d9f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.112+0000 7f1dbd7ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.112+0000 7f1dbd7ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.112+0000 7f1dbd7ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-29T11:30:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.112+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d98005330 con 0x7f1db81457f0 2024-03-29T11:30:55.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.113+0000 7f1d9f7fe640 1 -- 172.21.15.52:0/1582443233 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1539633666) 0x7f1dac04ef00 con 0x7f1db81457f0 2024-03-29T11:30:55.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.113+0000 7f1d9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.113+0000 7f1d9f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:55.108 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-29T11:30:55.108 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"3b9c51c5-3002-4ea5-be53-d626ccc5bde4","created":"2024-03-29T11:19:57.004209+0000","modified":"2024-03-29T11:30:01.922470+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:01.922470+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae9f13d7-d829-45b6-99ab-f3024ff4f4fb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b7490542-1dbf-4ecd-96e0-1008fc1c34a2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7dfc37ef-c9dd-4263-9eab-c406f16723c0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ecaf2963-5d88-405a-b213-5bca421a9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8dbe158-cae5-44f0-bbf3-dc1b5b3dbd48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c780a8b9-2eba-4d45-9693-5af6ad16610e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37614f81-d3d4-4fb9-b15f-1a4fe106d618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80826614-85ac-4fca-b118-837ebbdb3594","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:55.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 10 librados: watch_flush enter 2024-03-29T11:30:55.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 10 librados: watch_flush exit 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 10 client.4670.objecter successfully canceled tick 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 >> v1:172.21.15.79:6816/407480584 conn(0x7f1da406e130 legacy=0x7f1da40705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 10 monclient: shutdown 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.118+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 >> v1:172.21.15.52:6790/0 conn(0x7f1db81457f0 legacy=0x7f1db81e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.119+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 shutdown_connections 2024-03-29T11:30:55.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.119+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 >> 172.21.15.52:0/1582443233 conn(0x7f1db8066a00 msgr2=0x7f1db8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:55.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.120+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 shutdown_connections 2024-03-29T11:30:55.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.120+0000 7f1dbd7ef640 1 -- 172.21.15.52:0/1582443233 wait complete. 2024-03-29T11:30:55.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-29T11:30:55.120+0000 7f1dbd7ef640 1 librados: shutdown 2024-03-29T11:30:55.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:55.139 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:55.142 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:55.142 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:55.142 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.177 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:55.177 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:55.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.255 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:55.255 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:55.255 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.324 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:55.324 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:55.325 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.395 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:55.395 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:55.395 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.432 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:55.433 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:55.433 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.507 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:55.507 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:55.507 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.578 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:55.578 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:55.578 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.644 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:55.644 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:55.644 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:55.645 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.669 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:55.669 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:55.669 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.708 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:55.709 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:55.709 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:55.709 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.720 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:55.721 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:55.721 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.773 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:55.773 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:55.774 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:55.784 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:55.785 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:55.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:55.846 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:30:55.846 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:57.410 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:30:57.410 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:57.514 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:30:57.514 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:57.587 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:30:57.587 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:57.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:30:57.646 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:00.883 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:31:00.883 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:00.952 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:31:00.952 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:01.011 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:31:01.011 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:01.068 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:31:01.069 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/data/mon.a.tgz 2024-03-29T11:31:01.069 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-29T11:31:01.087 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.ftET9EefAj 2024-03-29T11:31:01.087 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ftET9EefAj 2024-03-29T11:31:01.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.ftET9EefAj 2024-03-29T11:31:01.279 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.ftET9EefAj is 126KB 2024-03-29T11:31:01.303 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.ftET9EefAj 2024-03-29T11:31:01.318 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/data/mon.c.tgz 2024-03-29T11:31:01.319 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-29T11:31:01.373 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.zs32ud17eA 2024-03-29T11:31:01.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zs32ud17eA 2024-03-29T11:31:01.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.zs32ud17eA 2024-03-29T11:31:01.548 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.zs32ud17eA is 126KB 2024-03-29T11:31:01.598 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.zs32ud17eA 2024-03-29T11:31:01.612 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/data/mon.b.tgz 2024-03-29T11:31:01.612 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-03-29T11:31:01.630 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.G7dc115eam 2024-03-29T11:31:01.631 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.G7dc115eam 2024-03-29T11:31:01.741 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.G7dc115eam 2024-03-29T11:31:01.830 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.G7dc115eam is 126KB 2024-03-29T11:31:01.858 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.G7dc115eam 2024-03-29T11:31:01.873 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:31:01.874 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:01.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:01.958 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:01.999 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:01.999 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:31:02.139 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/remote/smithi052/crash 2024-03-29T11:31:02.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:02.246 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/remote/smithi079/crash 2024-03-29T11:31:02.248 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:02.276 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:31:02.276 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:02.293 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34790.log 2024-03-29T11:31:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:31:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34910.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34790.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34630.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34910.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34790.log.gz 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34630.log: gzip -5 87.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.34910.log.gz /var/log/ceph/ceph-osd.4.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34870.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34630.log.gz 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.34710.log -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34870.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34670.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34870.log.gz 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.7.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34830.log 2024-03-29T11:31:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34670.log.gz 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34830.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34830.log.gz 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34750.log 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34750.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34750.log.gz 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34585.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39253.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr: 87.6%/var/log/ceph/ceph-client.admin.34585.log: -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38473.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34585.log.gz 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41614.log: /var/log/ceph/ceph-client.admin.38473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38473.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39175.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41222.log: 85.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz -5 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39175.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39175.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38902.log 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-03-29T11:31:02.371 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38902.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38551.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38902.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41105.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38512.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38551.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39934.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38512.log.gz 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39643.log 2024-03-29T11:31:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39370.log: gzip -5 --verbose 86.7% -- -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz /var/log/ceph/ceph-client.admin.39058.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38668.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39058.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39643.log.gz 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38236.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38668.log.gz 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38236.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38197.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38707.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38236.log.gz 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39565.log 2024-03-29T11:31:02.373 INFO:teuthology.orchestra.run.smithi052.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38707.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38197.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39565.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38707.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38590.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39565.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.audit.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40637.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38590.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-29T11:31:02.374 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40051.log: /var/log/ceph/ceph-client.admin.41575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40363.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28774.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41417.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.tmp-client.admin.28774.log: gzip 85.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz -- 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr: /var/log/ceph/ceph-client.admin.39331.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28774.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39331.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.1.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-03-29T11:31:02.375 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39526.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39487.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40481.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39526.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41456.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39698.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39487.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-03-29T11:31:02.376 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41456.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41456.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38863.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz/var/log/ceph/ceph-client.admin.38863.log: 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2%/var/log/ceph/ceph-client.admin.40676.log: -- replaced with /var/log/ceph/ceph-client.admin.38863.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39973.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz 2024-03-29T11:31:02.377 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39448.log: /var/log/ceph/ceph-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40441.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38824.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38941.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38276.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38824.log.gz 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38941.log.gz 2024-03-29T11:31:02.378 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39778.log 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38276.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38276.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40988.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39778.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34905.log 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41027.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.34905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39604.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34905.log.gz 2024-03-29T11:31:02.379 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41536.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-mgr.y.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39019.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40285.log: gzip -5 --verbose -- 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr: /var/log/ceph/ceph-client.admin.41183.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39019.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38393.log 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr:gzip/var/log/ceph/ceph-client.admin.41183.log: 86.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39019.log.gz -- 2024-03-29T11:31:02.380 INFO:teuthology.orchestra.run.smithi052.stderr: /var/log/ceph/ceph-client.admin.40090.log 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40090.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38354.log 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38393.log.gz 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40949.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log/var/log/ceph/ceph-client.admin.38354.log: 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38354.log.gz 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:31:02.381 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40207.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.2.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34979.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39214.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38785.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.34979.log: /var/log/ceph/ceph-client.admin.40793.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39817.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38432.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38785.log: /var/log/ceph/ceph-client.admin.39817.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38785.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39817.log.gz 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-03-29T11:31:02.382 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38432.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38432.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41497.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40129.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41497.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41497.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39409.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39856.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38629.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39409.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39409.log.gz 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-03-29T11:31:02.383 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38629.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39739.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38629.log.gz 2024-03-29T11:31:02.384 INFO:teuthology.orchestra.run.smithi052.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-03-29T11:31:02.427 INFO:teuthology.orchestra.run.smithi079.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:31:13.875 INFO:teuthology.orchestra.run.smithi079.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:31:13.891 INFO:teuthology.orchestra.run.smithi079.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:31:14.346 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:31:16.244 INFO:teuthology.orchestra.run.smithi052.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:31:16.806 INFO:teuthology.orchestra.run.smithi052.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:31:17.902 INFO:teuthology.orchestra.run.smithi079.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:31:17.987 INFO:teuthology.orchestra.run.smithi079.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:31:18.550 INFO:teuthology.orchestra.run.smithi052.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:31:52.690 INFO:teuthology.orchestra.run.smithi052.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:31:57.033 INFO:teuthology.orchestra.run.smithi052.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:31:57.035 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-29T11:31:57.035 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m54.728s 2024-03-29T11:31:57.035 INFO:teuthology.orchestra.run.smithi052.stderr:user 2m36.195s 2024-03-29T11:31:57.035 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m5.206s 2024-03-29T11:32:09.422 INFO:teuthology.orchestra.run.smithi079.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:32:09.423 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T11:32:09.423 INFO:teuthology.orchestra.run.smithi079.stderr:real 1m7.082s 2024-03-29T11:32:09.423 INFO:teuthology.orchestra.run.smithi079.stderr:user 1m56.814s 2024-03-29T11:32:09.423 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m4.293s 2024-03-29T11:32:09.424 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:32:09.425 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/remote/smithi052/log 2024-03-29T11:32:09.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:32.571 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/remote/smithi079/log 2024-03-29T11:32:32.572 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:52.348 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:52.571 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=324e8a2d9194420da95cdd9c547b581e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:52.579 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:52.607 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:52.621 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:52.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:52.638 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:52.714 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:52.734 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:52.735 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:52.738 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:52.752 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:52.755 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:52.756 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:52.756 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-29T11:32:52.757 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +475us[ +475us] +/- 92ms 2024-03-29T11:32:52.757 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -89us[ -90us] +/- 55ms 2024-03-29T11:32:52.757 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -251us[ -251us] +/- 86ms 2024-03-29T11:32:52.757 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:52.760 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:52.760 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-29T11:32:52.760 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +463us[ +455us] +/- 92ms 2024-03-29T11:32:52.761 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -115us[ -123us] +/- 55ms 2024-03-29T11:32:52.761 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -267us[ -267us] +/- 86ms 2024-03-29T11:32:52.761 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:52.762 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:52.774 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:52.777 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:52.793 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:52.872 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:52.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:52.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:53.007 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:53.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:53.039 INFO:teuthology.task.internal:Duration was 1347.065792 seconds 2024-03-29T11:32:53.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:53.056 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:53.056 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:53.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:53.103 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:53.107 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:53.403 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:53.404 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-03-29T11:32:53.404 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:53.425 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:32:53.425 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:53.458 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:53.458 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:53.468 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:53.530 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:53.531 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:53.541 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:53.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:53.785 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:53.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:53.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:53.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:53.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:54.001 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:54.024 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-03-29T11:32:54.027 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-03-29T11:32:54.051 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:54.109 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:54.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:54.187 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:54.214 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/remote/smithi052 2024-03-29T11:32:54.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:54.263 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695/remote/smithi079 2024-03-29T11:32:54.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:54.310 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:54.310 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:54.313 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:54.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:54.363 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:54.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:54.398 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:54.399 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:54.401 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:54.413 INFO:teuthology.orchestra.run.smithi052.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:54.416 INFO:teuthology.orchestra.run.smithi079.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:54.417 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:54.429 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:54.471 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:54.498 DEBUG:teuthology.nuke:shortname: smithi052 2024-03-29T11:32:54.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:54.520 DEBUG:teuthology.nuke:shortname: smithi079 2024-03-29T11:32:54.521 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:54.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.598261', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbbdhHQxBQkJRN769eMBmzh/0mCvr4Sar+m7eLmjdJTrB7PzcV4p69K2BYniUODFRvANxG2lols4JsHnC2ZZ3Q='} 2024-03-29T11:32:54.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622695', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.596330', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNA5jl3+PWU8sr0KWl1Dye4j9g+m0x3sFQfImSX++M/iQdOsHG4erVwWf4OzwDEZKro2PkTqEYhDkrEndVGrY+Y='} 2024-03-29T11:32:54.577 INFO:teuthology.orchestra.console.smithi052:Power off 2024-03-29T11:32:54.578 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:54.606 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:54.606 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:54.625 INFO:teuthology.orchestra.console.smithi079:Power off 2024-03-29T11:32:54.625 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:54.635 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-29T11:32:54.648 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:54.649 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:54.672 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-29T11:32:58.636 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:58.673 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:58.748 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-29T11:32:58.783 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-29T11:33:02.754 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:02.784 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:02.865 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-03-29T11:33:02.895 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-03-29T11:33:02.966 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-03-29T11:33:02.997 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-03-29T11:33:03.101 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1347.0657920837402 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=324e8a2d9194420da95cdd9c547b581e status: fail success: false 2024-03-29T11:33:03.101 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:33:03.178 INFO:teuthology.run:FAIL